[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-27 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/14079 I left a bunch of cosmetic comments to try to make this code as easy as possible to read / reason about. Taking a break for a meeting but will finish reviewing this later this afternoon

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62894/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62894/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62894/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62737/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62737/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62737/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62734/ Test FAILed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62734/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 I updated it to take into account preemption, and also noticed that `TaskKilled` and `TaskCommitDenied` weren't handled either, so I added those and a test. I also tried to clean up some of the

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-22 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62734/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62630/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62630/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62630/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62621/ Test FAILed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62621/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62621/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 thanks for the feedback @tgravescs . Thanks for pointing out the issue after executors have died, I updated the behavior and added a test case (I'll update the design doc shortly too). I think I

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14079 this can probably be a separate jira but it would be really nice to show blacklisted on executors page when it happens. i guess it can get a bit complicated with timeout from blacklist and

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62352/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62352/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62351/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62351 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62351/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62352/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62351 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62351/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14079 Just some minor stuff, I'll let people more familiar with the scheduler comment further. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62253/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62253 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62253/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62245/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62245/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62253 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62253/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-13 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62245/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62134/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62134/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62134/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 discussed this offline with @vanzin, realized that actually with the latest design, it doesn't make sense to have so many maps inside one `BlacklistTracker` -- a lot of info is now intended to only

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62119/ Test FAILed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62119/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-11 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #62119 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62119/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61931/ Test FAILed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #61931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61931/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-07 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #61931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61931/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-07 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 I took another look at having BlacklistTracker just be an option, rather than having a NoopBlacklist. After some other cleanup, I decided it made more sense to go back to the option, but its in one

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61873/ Test PASSed. ---

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #61873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61873/consoleFull)** for PR 14079 at commit

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-06 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/14079 @kayousterhout @markhamstra @tgravescs @mwws I finally this is ready for review. I have some minor updates left but I wanted to get this in your hands now. The main thing is testing on a cluster

[GitHub] spark issue #14079: [SPARK-8425][CORE] New Blacklist Mechanism

2016-07-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14079 **[Test build #61873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61873/consoleFull)** for PR 14079 at commit

<    1   2