[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153117330 Hi @kayousterhout , are you ok with my explanation above? I'd like to get this is soon. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153117844 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153119411 **[Test build #44819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44819/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153117825 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r43670488 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -468,11 +468,18 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153134683 Scheduler changes LGTM, with the documentation improvements I suggested --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153138287 **[Test build #44828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44828/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153137357 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153137327 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153152212 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153152206 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r43670630 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -482,7 +489,10 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153151863 **[Test build #44819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44819/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153166216 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153165888 hmm, weird failure. retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153170036 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153170034 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153164574 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153164573 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153172694 really jenkins, help me out here. retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153164500 **[Test build #44828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44828/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153173976 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153166264 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153173958 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153176554 **[Test build #44841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44841/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153201756 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153201753 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-11-02 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-153201676 **[Test build #44841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44841/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152790319 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152790196 **[Test build #44744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44744/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152790318 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152784361 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152784357 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-152785029 **[Test build #44744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44744/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150992179 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150992191 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42952922 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -237,4 +237,39 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42952925 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -237,4 +237,39 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150992356 So, my code was slightly different, and that caused the problem with my unit test; but I still think my change is more correct. Not only it avoids unnecessary error logs

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150991709 Huh, there's more code in my approach because with just the code you had, the unit test I added failed. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-151006183 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-151006134 **[Test build #44323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44323/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-151006185 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150991951 What was the problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150991608 I still prefer the approach in the commit I submitted, because it puts all of the code to handle the pending loss reasons in one place in TaskSchedulerImpl

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-25 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150993653 **[Test build #44323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44323/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-23 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150657909 So, any remaining feedback? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150072395 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150072364 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150075686 **[Test build #44117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44117/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150099051 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150099052 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-150098979 **[Test build #44117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44117/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42439726 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -482,21 +482,28 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149377970 The coraseGrainedSchedulerBackend changes are fine; I meant the changes to TaskSchedulerImpl / TaskSetManager. What do you mean that my change immediately marks

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42438520 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -482,21 +482,28 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149376270 The scheduler changes still seem more complicated than what I proposed. Why not do this the simpler way? --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149379158 Derp yeah that !(==) was weird! The recomputeLocality() call seems like it will matter very rarely, so I think in the name of simplicity we should skip it. ---

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149380248 I had to remove the unit test because that test cannot call `TaskSchedulerImpl.executorLost`, which is the only place that's changing now... --- If your project is set

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149373071 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149373092 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149378632 True; I think the weird `!( == )` comparison tricked me. But your change also doesn't call `recomputeLocality()` for the first `executorLost` call, which could make a

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149381675 > proposing adding a test to TaskSchedulerImplSuite I did, but last time I looked at that test suite I didn't have good luck in figuring out how to add a useful

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42440043 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -482,21 +482,28 @@ private[spark] class TaskSchedulerImpl( }

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149380561 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149380583 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149380773 Did you see my comment proposing adding a test to TaskSchedulerImplSuite? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42441070 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -73,6 +73,10 @@ class

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149381337 **[Test build #43955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43955/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149397569 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149397539 **[Test build #43955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43955/consoleFull)** for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-149397568 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148865840 (BTW that's what the unit test I added was meant to test.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148865650 > So what do those calls do? My understanding is that you may have pending tasks waiting to be run on that executor (because of PROCESS_LOCAL locality?

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148863767 I think the dag scheduler still gets updated right away. You're right that some tasks might be stuck in the TaskSetManager -- I interpreted your goal as being

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148865197 Sorry, you're right, in your cxode the dag scheduler is updated right away. Only the task set manager is not. The delay is hopefully not that long. As part of

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148867465 Just submitted https://github.com/apache/spark/pull/9154 because I think those calls are unnecessary...but let's wait for Mark to weigh in there to make sure.

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148862564 @kayousterhout but that's not equivalent to my change, is it? My change is more proactive in updating the scheduler's state to reflect that the executor doesn't exist

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148857831 @vanzin I was thinking of something much simpler, like this: https://github.com/kayousterhout/spark-1/commit/da26fa02d57ca12f7e50105930ca46c390aa3098 (with this

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148864516 Actually I'm confused: with your change, when the ExecutorLost message is received, you call "addPendingTask" again for any still-pending tasks for that executor

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42266241 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala --- @@ -37,6 +37,8 @@ private[spark] object ExecutorExited { } }

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148588048 @andrewor14 I think @vanzin was waiting on me to give the thumb's up for the original approach (not the current diff) but I haven't had a chance to look at this

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42198446 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala --- @@ -37,6 +37,8 @@ private[spark] object ExecutorExited { }

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42199327 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -73,6 +73,10 @@ class

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148552709 [Test build #43810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43810/console) for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148552806 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148552808 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42196645 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -362,8 +361,8 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42198351 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -459,46 +458,64 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42198578 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -795,31 +795,40 @@ private[spark] class TaskSetManager(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42199201 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala --- @@ -776,6 +776,30 @@ class TaskSetManagerSuite extends

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42199262 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -249,6 +258,30 @@ class

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42196018 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -325,16 +325,15 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/8887#discussion_r42195991 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -362,8 +395,8 @@ private[spark] class TaskSchedulerImpl(

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148526821 [Test build #43810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43810/consoleFull) for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148525757 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-148525781 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-13 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-147908559 [Test build #43697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43697/consoleFull) for PR 8887 at commit

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-13 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-147907985 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-10622] [core] [yarn] Differentiate dead...

2015-10-13 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8887#issuecomment-147907083 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

  1   2   >