Repository: spark Updated Branches: refs/heads/master f7cf2096f -> 640afa49a
[SPARK-20365][YARN] Remove local scheme when add path to ClassPath. In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local" scheme), we will get inaccurate classpath for AM and containers. This is because we don't remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme. Updated `ClientSuite` to check "local" is not in the classpath. cc jerryshao Author: Li Yichao <[email protected]> Author: Li Yichao <[email protected]> Closes #18129 from liyichao/SPARK-20365. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/640afa49 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/640afa49 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/640afa49 Branch: refs/heads/master Commit: 640afa49aa349c7ebe35d365eec3ef9bb7710b1d Parents: f7cf209 Author: Li Yichao <[email protected]> Authored: Thu Jun 1 14:39:57 2017 -0700 Committer: Marcelo Vanzin <[email protected]> Committed: Thu Jun 1 14:40:05 2017 -0700 ---------------------------------------------------------------------- .../yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala | 3 ++- .../src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/640afa49/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---------------------------------------------------------------------- diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala index 9956071..1fb7edf 100644 --- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala +++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala @@ -1275,7 +1275,8 @@ private object Client extends Logging { if (sparkConf.get(SPARK_ARCHIVE).isEmpty) { sparkConf.get(SPARK_JARS).foreach { jars => jars.filter(isLocalUri).foreach { jar => - addClasspathEntry(getClusterPath(sparkConf, jar), env) + val uri = new URI(jar) + addClasspathEntry(getClusterPath(sparkConf, uri.getPath()), env) } } } http://git-wip-us.apache.org/repos/asf/spark/blob/640afa49/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---------------------------------------------------------------------- diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala index 3a11787..6cf6842 100644 --- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala +++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala @@ -122,6 +122,7 @@ class ClientSuite extends SparkFunSuite with Matchers with BeforeAndAfterAll cp should not contain (uri.getPath()) } }) + cp should not contain ("local") cp should contain(PWD) cp should contain (s"$PWD${Path.SEPARATOR}${LOCALIZED_CONF_DIR}") cp should not contain (APP_JAR) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
