[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082129#comment-17082129 ]
Shingo Furuyama commented on SPARK-31432: ----------------------------------------- I will soon send a PR to the master branch. If the PR is merged, I will send it branch-2.4. > bin/sbin scripts should allow to customize jars dir > --------------------------------------------------- > > Key: SPARK-31432 > URL: https://issues.apache.org/jira/browse/SPARK-31432 > Project: Spark > Issue Type: Improvement > Components: Deploy > Affects Versions: 2.4.5, 3.0.0 > Reporter: Shingo Furuyama > Priority: Minor > > In the script under bin/sbin, it is better that we can specify SPARK_JARS_DIR > as same as SPARK_CONF_DIR. > Our usecase: > We are trying to employ spark 2.4.5 with YARN in HDP2.6.4. Since there is an > incompatible conflict on commons-lang3 between spark 2.4.5 and HDP2.6.4, we > tweak the jars by Maven Shade Plugin. > The jars slightly differ from jars in spark 2.4.5, and we locate it in a > directory different from the default. So it is useful for us if we can set > SPARK_JARS_DIR for bin/sbin scripts to point the direcotry. > We can do that without the modification by deploying spark home as many as > set of jars, but it is somehow redundant. > Common usecase: > I believe there is a similer usecase. For example, deploying spark built for > scala 2.11 and 2.12 in a machine and switch jars location by setting > SPARK_JARS_DIR. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org