[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim closed the pull request at: https://github.com/apache/spark/pull/1516 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-50313978 SPARK-2567 is resolved by #1566. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49845710 How does this affect the final UI ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49909562 You can see the screenshot which the original code generated in the JIRA. https://issues.apache.org/jira/browse/SPARK-2567 This screenshot was taken after the job completed but one stage remained as Active Stage forever. It shouldn't be displayed in this web ui at all because the corresponding new TaskSet is not submitted and even stage.newAttemptId() isn't called. Sometimes this ghost stage appears when stage is re-submitted so this PR modified to prevent web ui from showing it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49911032 This appears to be a reversion of d58502a1562bbfb1bb4e517ebcc8239efd639297 while ignoring and misapplying the comment regarding ordering (which I'm not completely understanding.) @xiajunluan ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49919528 Hmm... I didn't notice it. I'm going to rerun the test for confirmation as @xiajunluan 's commit comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49926056 Actually I aim to fix this in https://github.com/apache/spark/pull/1545 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49940152 The test totally succeeded again. If the @xiajunluan 's commit only aimed to avoid the unit test error, I think it should be reversioned as this PR. But I'm wondering if there were another aim. @xiajunluan could you remember it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user tsudukim commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49940743 Hi @rxin, thank you for following this ticket but couldn't we separate those problems into different PRs? SPARK-2298 is not about this problem. I think we will be hard to trace why the code was modified and what discussion was made on the topic later. (just like we are now wondering about the intention of the commit of the last year) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49942454 Actually the fix looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49942459 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49942652 QA tests have started for PR 1516. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17063/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49950833 QA results for PR 1516:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17063/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49968180 I looked into this a bit more and this change doesn't quite fix the problem in the right way, partially because of what @markhamstra pointed out (that we shouldn't send a SparkListenerStageCompleted event before sending a corresponding SparkListenerStageSubmitted event, which is I think what the mysterious comment was getting at) and partially because the bigger underlying problem here is that runningStages isn't updated at the right time (which also leads to a memory leak). I submitted an alternate fix here: https://github.com/apache/spark/pull/1566 -- let me know what you all think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1516#issuecomment-49666932 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2567] Resubmitted stage sometimes remai...
GitHub user tsudukim opened a pull request: https://github.com/apache/spark/pull/1516 [SPARK-2567] Resubmitted stage sometimes remains as active stage in the web UI Moved the line which post SparkListenerStageSubmitted to the back of check of tasks size and serializability. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tsudukim/spark feature/SPARK-2567 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1516.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1516 commit 79f4f98f5d08b7f52bb0216f2b412d959e64ad89 Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Date: 2014-07-21T21:05:42Z [SPARK-2567] Resubmitted stage sometimes remains as active stage in the web UI Moved the line which post SparkListenerStageSubmitted to the back of check of tasks size and serializability. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
