[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76089/ Test PASSed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #76089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76089/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #76089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76089/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76076/ Test FAILed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #76076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76076/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-19 Thread witgo
Github user witgo commented on the issue: https://github.com/apache/spark/pull/17480 OK, I will do the work at weekends. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-18 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17480 I probably won't have time to look at a proper fix for this anytime soon, but I don't think your current patch is the right fix. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-18 Thread witgo
Github user witgo commented on the issue: https://github.com/apache/spark/pull/17480 @vanzin Sorry, I do not understand what you mean. Do you submit a new PR to your own ideas? If you can, I will close this PR. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-17 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17480 @witgo are you planning to update this PR to fix the behavior of `reset` in call cases? The biggest problem I have with this patch is that reading the code does not give you any insight

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17480 Sorry but that doesn't really explain much. Why is it bad to ramp up quickly? At which point are things not "initializing" anymore? Isn't the AM restarting the definition of "I should ramp

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75601/ Test PASSed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75601/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75601/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-05 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17480 Also CC @tgravescs @vanzin to help to review, they may have more thoughts :). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75508/ Test PASSed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75508 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75508/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75508 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75508/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75499/ Test FAILed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-04-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75499/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-31 Thread witgo
Github user witgo commented on the issue: https://github.com/apache/spark/pull/17480 @jerryshao Yes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-31 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17480 @witgo thanks for your explanation. But AFAIK if AM get restarted, it will honor initial executor number to launch executors, so after executors are launched, stage should be able to get

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-31 Thread witgo
Github user witgo commented on the issue: https://github.com/apache/spark/pull/17480 The ExecutorAllocationManager.reset method is called when re-registering AM, which sets the ExecutorAllocationManager.initializing field true. When this field is true, the Driver does not start a new

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-30 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17480 Would you please help to elaborate the problem you met? That would be better to understand your scenario and fix. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75391/ Test PASSed. ---

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17480 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75391/testReport)** for PR 17480 at commit

[GitHub] spark issue #17480: [SPARK-20079][Core][yarn] Re registration of AM hangs sp...

2017-03-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17480 **[Test build #75391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75391/testReport)** for PR 17480 at commit