GitHub user aarondav opened a pull request:
https://github.com/apache/spark/pull/1409
SPARK-1097: Do not introduce deadlock while fixing concurrency bug
We recently added this lock on 'conf' in order to prevent concurrent
creation. However, it turns out that this can introduce a deadlock because
Hadoop also synchronizes on the Configuration objects when creating new
Configurations (and they do so via a static REGISTRY which contains all created
Configurations).
This fix forces all Spark initialization of Configuration objects to occur
serially by using a static lock that we control, and thus also prevents
introducing the deadlock.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aarondav/spark 1054
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1409.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1409
----
commit 7d1b769f943bdc35a410e0225584b047602f68f4
Author: Aaron Davidson <[email protected]>
Date: 2014-07-14T21:58:58Z
SPARK-1097: Do not introduce deadlock while fixing concurrency bug
We recently added this lock on 'conf' in order to prevent concurrent
creation. However, it turns out that this can introduce a deadlock
because Hadoop also synchronizes on the Configuration objects when
creating new Configurations (and they do so via a static REGISTRY
which contains all created Configurations).
This fix forces all Spark initialization of Configuration objects
to occur serially by using a static lock that we control, and thus
also prevents introducing the deadlock.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
