Repository: spark Updated Branches: refs/heads/branch-2.0 1145ea01b -> eb7336a75
[SPARK-14845][SPARK_SUBMIT][YARN] spark.files in properties file is n⦠## What changes were proposed in this pull request? initialize SparkSubmitArgument#files first from spark-submit arguments then from properties file, so that sys property spark.yarn.dist.files will be set correctly. ``` OptionAssigner(args.files, YARN, ALL_DEPLOY_MODES, sysProp = "spark.yarn.dist.files"), ``` ## How was this patch tested? manul test. file defined in properties file is also distributed to driver in yarn-cluster mode. Author: Jeff Zhang <[email protected]> Closes #12656 from zjffdu/SPARK-14845. (cherry picked from commit 0a3026990bd0cbad53f0001da793349201104958) Signed-off-by: Marcelo Vanzin <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eb7336a7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eb7336a7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eb7336a7 Branch: refs/heads/branch-2.0 Commit: eb7336a754574879fc28c3c10cb68651329a346d Parents: 1145ea0 Author: Jeff Zhang <[email protected]> Authored: Mon May 2 11:03:37 2016 -0700 Committer: Marcelo Vanzin <[email protected]> Committed: Mon May 2 11:03:49 2016 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 1 + 1 file changed, 1 insertion(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/eb7336a7/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala index 78da1b7..206c130 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala @@ -173,6 +173,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S .orNull name = Option(name).orElse(sparkProperties.get("spark.app.name")).orNull jars = Option(jars).orElse(sparkProperties.get("spark.jars")).orNull + files = Option(files).orElse(sparkProperties.get("spark.files")).orNull ivyRepoPath = sparkProperties.get("spark.jars.ivy").orNull packages = Option(packages).orElse(sparkProperties.get("spark.jars.packages")).orNull packagesExclusions = Option(packagesExclusions) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
