GitHub user zjffdu reopened a pull request:
https://github.com/apache/spark/pull/15669
[SPARK-18160][CORE][YARN] spark.files should not be passed to driver in
yarn-cluster mode
## What changes were proposed in this pull request?
spark.files is still passed to driver in yarn mode, so SparkContext will
still handle it which cause the error in the jira desc.
## How was this patch tested?
Tested manually in a 5 node cluster. As this issue only happens in multiple
node cluster, so I didn't write test for it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/spark SPARK-18160
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15669.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15669
----
commit 67a5ccf7a9d02a8b930ab97e10c0858b4d046496
Author: Jeff Zhang <[email protected]>
Date: 2016-10-28T07:50:59Z
[SPARK-18160][CORE][YARN] SparkContext.addFile doesn't work in yarn-cluster
mode
commit 8033bd1bce9aa2fa0b05fe53c66ea072656dbd23
Author: Jeff Zhang <[email protected]>
Date: 2016-10-31T23:23:53Z
Revert "[SPARK-18160][CORE][YARN] SparkContext.addFile doesn't work in
yarn-cluster mode"
This reverts commit 67a5ccf7a9d02a8b930ab97e10c0858b4d046496.
commit 230d56c90ce7ff30251a03afe6c677fe9df8faca
Author: Jeff Zhang <[email protected]>
Date: 2016-11-01T03:58:08Z
remove spark.files & spark.jars from SparkConf in yarn mode
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
