[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49513420 @aarondav @pwendell In my tests, it seems that there are still a deadlock. To find a possible reason this here [Executor.scala#L189] (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L189) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49204042 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49204801 QA tests have started for PR 1409. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16740/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49218047 QA results for PR 1409:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16740/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49227548 Okay I'm going to merge this into master and 1.0. We can cut a new patch release shortly for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1409 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user koeninger commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-49066268 Testing that patch, it seems to have fixed the deadlock we were seeing in production. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/1409 SPARK-1097: Do not introduce deadlock while fixing concurrency bug We recently added this lock on 'conf' in order to prevent concurrent creation. However, it turns out that this can introduce a deadlock because Hadoop also synchronizes on the Configuration objects when creating new Configurations (and they do so via a static REGISTRY which contains all created Configurations). This fix forces all Spark initialization of Configuration objects to occur serially by using a static lock that we control, and thus also prevents introducing the deadlock. You can merge this pull request into a Git repository by running: $ git pull https://github.com/aarondav/spark 1054 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1409 commit 7d1b769f943bdc35a410e0225584b047602f68f4 Author: Aaron Davidson aa...@databricks.com Date: 2014-07-14T21:58:58Z SPARK-1097: Do not introduce deadlock while fixing concurrency bug We recently added this lock on 'conf' in order to prevent concurrent creation. However, it turns out that this can introduce a deadlock because Hadoop also synchronizes on the Configuration objects when creating new Configurations (and they do so via a static REGISTRY which contains all created Configurations). This fix forces all Spark initialization of Configuration objects to occur serially by using a static lock that we control, and thus also prevents introducing the deadlock. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1097: Do not introduce deadlock while fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1409#issuecomment-48966606 QA tests have started for PR 1409. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16639/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
