[GitHub] spark pull request #15669: [SPARK-18160][CORE][YARN] spark.files should not ...
Github user zjffdu closed the pull request at: https://github.com/apache/spark/pull/15669 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15669: [SPARK-18160][CORE][YARN] spark.files should not ...
GitHub user zjffdu reopened a pull request: https://github.com/apache/spark/pull/15669 [SPARK-18160][CORE][YARN] spark.files should not be passed to driver in yarn-cluster mode ## What changes were proposed in this pull request? spark.files is still passed to driver in yarn mode, so SparkContext will still handle it which cause the error in the jira desc. ## How was this patch tested? Tested manually in a 5 node cluster. As this issue only happens in multiple node cluster, so I didn't write test for it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/spark SPARK-18160 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15669.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15669 commit 67a5ccf7a9d02a8b930ab97e10c0858b4d046496 Author: Jeff ZhangDate: 2016-10-28T07:50:59Z [SPARK-18160][CORE][YARN] SparkContext.addFile doesn't work in yarn-cluster mode commit 8033bd1bce9aa2fa0b05fe53c66ea072656dbd23 Author: Jeff Zhang Date: 2016-10-31T23:23:53Z Revert "[SPARK-18160][CORE][YARN] SparkContext.addFile doesn't work in yarn-cluster mode" This reverts commit 67a5ccf7a9d02a8b930ab97e10c0858b4d046496. commit 230d56c90ce7ff30251a03afe6c677fe9df8faca Author: Jeff Zhang Date: 2016-11-01T03:58:08Z remove spark.files & spark.jars from SparkConf in yarn mode --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15669: [SPARK-18160][CORE][YARN] spark.files should not ...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/15669#discussion_r85872343 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1716,29 +1716,12 @@ class SparkContext(config: SparkConf) extends Logging { key = uri.getScheme match { // A JAR file which exists only on the driver node case null | "file" => -if (master == "yarn" && deployMode == "cluster") { - // In order for this to work in yarn cluster mode the user must specify the - // --addJars option to the client to upload the file into the distributed cache - // of the AM to make it show up in the current working directory. - val fileName = new Path(uri.getPath).getName() - try { -env.rpcEnv.fileServer.addJar(new File(fileName)) - } catch { -case e: Exception => - // For now just log an error but allow to go through so spark examples work. - // The spark examples don't really need the jar distributed since its also - // the app jar. - logError("Error adding jar (" + e + "), was the --addJars option used?") - null - } -} else { - try { -env.rpcEnv.fileServer.addJar(new File(uri.getPath)) - } catch { -case exc: FileNotFoundException => - logError(s"Jar not found at $path") - null - } --- End diff -- These are obsoleted code IMO, so I remove them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org