Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13527#discussion_r66000637
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -444,6 +444,7 @@ object SparkSubmit {
OptionAssigner(args.deployMode, ALL_CLUSTER_MGRS, ALL_DEPLOY_MODES,
sysProp = "spark.submit.deployMode"),
OptionAssigner(args.name, ALL_CLUSTER_MGRS, ALL_DEPLOY_MODES,
sysProp = "spark.app.name"),
+ OptionAssigner(args.jars, ALL_CLUSTER_MGRS, CLIENT, sysProp =
"spark.jars"),
--- End diff --
> How should Spark behave when a user both sets spark.jars and
spark.yarn.dist.jars? Do we want to set union them?
Yes, I think that should be the expected behavior. There might be some
weird semantics that need to be addressed, though:
- user sets `spark.jars` in conf
- SparkSubmitArguments sets `args.jars` to value of `spark.jars`
- YARN-specific code sets `spark.yarn.dist.jars` to value of `args.jars`
- Now both `spark.jars` and `spark.yarn.dist.jars` have the same value.
Perhaps when the YARN translation happens above, `spark.jars` should be
explicitly unset.
> In SparkConf there is a setJars() method that sets spark.jars
I think it's ok to leave that as is. It could be done as an optimization,
but it's not necessarily related to this bug so let's leave that for a
different change, if desired.
> Does implementing option 2 resolve the issue completely?
I did some quick testing and it seems to work. You just need to make sure
the code works regardless of which option is set. The classpath for the repl
should be the concatenation of both options.
Optimally the repl would only look at the YARN-specific options when
running in YARN mode, but I think that code runs too early in the process for
Spark to already know whether it will use YARN or not.
Also remember you need to change both the 2.10 and 2.11 repl code. Might be
easier if you make it a helper method in `Utils.scala`, for example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
