[
https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-31432:
------------------------------------
Assignee: Apache Spark
> Make SPARK_JARS_DIR configurable
> --------------------------------
>
> Key: SPARK-31432
> URL: https://issues.apache.org/jira/browse/SPARK-31432
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Affects Versions: 3.1.0
> Reporter: Shingo Furuyama
> Assignee: Apache Spark
> Priority: Minor
>
> In the script under bin/sbin, it is better that we can specify SPARK_JARS_DIR
> as same as SPARK_CONF_DIR.
> Our usecase:
> We are trying to employ spark 2.4.5 with YARN in HDP2.6.4. Since there is an
> incompatible conflict on commons-lang3 between spark 2.4.5 and HDP2.6.4, we
> tweak the jars by Maven Shade Plugin.
> The jars slightly differ from jars in spark 2.4.5, and we locate it in a
> directory different from the default. So it is useful for us if we can set
> SPARK_JARS_DIR for bin/sbin scripts to point the direcotry.
> We can do that without the modification by deploying spark home as many as
> set of jars, but it is somehow redundant.
> Common usecase:
> I believe there is a similer usecase. For example, deploying spark built for
> scala 2.11 and 2.12 in a machine and switch jars location by setting
> SPARK_JARS_DIR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
