Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3233#discussion_r22975789
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -762,46 +764,37 @@ object Client extends Logging {
extraClassPath: Option[String] = None): Unit = {
extraClassPath.foreach(addClasspathEntry(_, env))
addClasspathEntry(Environment.PWD.$(), env)
-
- // Normally the users app.jar is last in case conflicts with spark jars
if (sparkConf.getBoolean("spark.yarn.user.classpath.first", false)) {
- addUserClasspath(args, sparkConf, env)
- addFileToClasspath(sparkJar(sparkConf), SPARK_JAR, env)
- populateHadoopClasspath(conf, env)
- } else {
- addFileToClasspath(sparkJar(sparkConf), SPARK_JAR, env)
- populateHadoopClasspath(conf, env)
- addUserClasspath(args, sparkConf, env)
+ getUserClasspath(args, sparkConf).foreach { x =>
+ addFileToClasspath(x, null, env)
+ }
}
-
- // Append all jar files under the working directory to the classpath.
- addClasspathEntry(Environment.PWD.$() + Path.SEPARATOR + "*", env)
--- End diff --
It's removed for two reasons:
- It didn't serve any practical purpose
- It could potentially lead to behavior that diverged from other cluster
managers
All jars distributed with `--jars` are added to the classpath
automatically, withouth the need for this. The directory itself is also added,
so things like `log4j.properties` uploaded by the user are in the classpath.
The only change this causes is that files and archives (`--files` and
`--archives`) would also end up in the app's classpath. This is the part that
diverges from other cluster managers - if you use `--files` to add a jar file
in standalone mode, the classes in that jar will not show up in the app's
classpath. In Yarn mode they would, and I think that's wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
