GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2684
[SPARK-2546] Clone JobConf for each task (branch-1.0 / 1.1 backport)
This patch attempts to fix SPARK-2546 in `branch-1.0` and `branch-1.1`.
The underlying problem is that thread-safety issues in Hadoop Configuration
objects may cause Spark tasks to get stuck in infinite loops. The approach
taken here is to clone a new copy of the JobConf for each task rather than
sharing a single copy between tasks. Note that there are still Configuration
thread-safety issues that may affect the driver, but these seem much less
likely to occur in practice and will be more complex to fix (see discussion on
the SPARK-2546 ticket).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark jobconf-fix-backport
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2684.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2684
----
commit dd25697c490e40f644b544c975afff49e107ace6
Author: Josh Rosen <[email protected]>
Date: 2014-10-06T23:26:29Z
[SPARK-2546] [1.0 / 1.1 backport] Clone JobConf for each task.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
