[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-25 Thread ilganeli
Github user ilganeli closed the pull request at: https://github.com/apache/spark/pull/4703 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-23 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-85319636 Hey @ilganeli would you mind closing this one since reviews are going on in the newer one? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-09 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-77930465 Hi @srowen and @markhamstra , with the exception of the refactoring for newResultStage that Mark proposed, I've added the recommended changes. Is there anything else

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-04 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-77263660 Hi all - I had to revert to my initial implementation since Mark's suggested refactoring introduced a test failure. Is this good to go? --- If your project is set up

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-04 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-77267879 Interesting. Looks like the failed test results are no longer available. Do you recall what the problem was? --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-03-04 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-77272127 It was a pretty obscure error. I could revert and give you the stack trace but I played around with it a bit and wasn't able to trace it down. --- If your project is

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75829212 Hi Sean - I'll give the retest a shot. If there's a way to get whitelisted that would be amazing. Thanks! --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75838380 [Test build #27900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27900/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75828243 [Test build #27900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27900/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75838391 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75824748 Retest please :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75827924 Looks like a transient git failure yes. I wonder if you have the power to tell Jenkins to retest? retest this please seems to work for me. If not maybe we can get you

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75855424 [Test build #27911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27911/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75865162 [Test build #27912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27912/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75856293 [Test build #27912 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27912/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75865176 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75881056 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75881048 [Test build #27917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27917/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75864145 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75864128 [Test build #27911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27911/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75855242 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75868482 [Test build #27917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27917/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75593300 [Test build #27854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27854/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread ilganeli
Github user ilganeli commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25183148 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -830,39 +836,39 @@ class DAGScheduler( try { // For

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75637201 Look pretty good to me, but left a few more comments. Also, please take a look at the various logging strings to see whether some of them can be expressed more

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25202931 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -210,40 +210,58 @@ class DAGScheduler( * The jobId value

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25186634 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -77,52 +71,9 @@ private[spark] class Stage( /** Pointer to the latest

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75604216 [Test build #27858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27858/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75609741 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75610748 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25186416 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -47,26 +47,20 @@ import org.apache.spark.util.CallSite * be updated for

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75594276 [Test build #27856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27856/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25186558 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -77,52 +71,9 @@ private[spark] class Stage( /** Pointer to the latest

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25188257 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -228,22 +227,41 @@ class DAGScheduler( } /** -

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75609726 [Test build #27854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27854/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75610731 [Test build #27856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27856/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75602215 @JoshRosen I'm happy to take a look at this but won't be able to get to it until Friday. Feel free to merge it sooner than that if you're eager to get it in;

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75620128 [Test build #27858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27858/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75620137 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread markhamstra
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75653958 nits: extra spaces in type declarations. E.g,: ```scala var numAvailableOutputs: Long = 0 ``` not ```scala var numAvailableOutputs : Long = 0

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75681861 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-23 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75691936 What happened with this test? Seems like it failed to fetch from git? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75492348 Hmm, it looks like MiMa is complaining about the `Stage` class becoming abstract even though it's `private[spark]`: ``` [info] spark-core: found 1

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75493744 I've added an exclude for the Spark 1.3 build - hopefully that resolves this issue. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145518 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -830,39 +836,39 @@ class DAGScheduler( try { // For

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145581 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -996,50 +1006,52 @@ class DAGScheduler( if

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145702 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ResultStage.scala --- @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145735 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ResultStage.scala --- @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75493784 [Test build #27850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27850/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145685 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapStage.scala --- @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4708#discussion_r25145446 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -830,39 +836,39 @@ class DAGScheduler( try { // For

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75494807 I took a quick pass, but overall this looks pretty good to me since it's inline with the refactoring that I had originally planned to do myself. It's nice to see a

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75499003 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75499000 [Test build #27850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27850/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25058069 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -484,7 +505,7 @@ class DAGScheduler( Total number of

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25055713 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -47,26 +47,19 @@ import org.apache.spark.util.CallSite * be updated for

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25056356 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -132,9 +88,7 @@ private[spark] class Stage( } def

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25057714 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -229,41 +227,56 @@ class DAGScheduler( /** *

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25057992 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -306,26 +319,31 @@ class DAGScheduler( } }

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25056104 --- Diff: core/src/main/scala/org/apache/spark/scheduler/Stage.scala --- @@ -77,53 +70,16 @@ private[spark] class Stage( /** Pointer to the latest

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25056762 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -229,41 +227,56 @@ class DAGScheduler( /** *

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/4703#discussion_r25058302 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -912,6 +959,196 @@ class DAGScheduler( } /** +

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75332520 [Test build #27793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27793/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-7531 [Test build #27788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27788/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75320629 Josh, what I'll do is create a new branch and port the changes individually and leave this branch intact as a reference. I'll submit a separate PR for that work. ---

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75318899 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread ilganeli
GitHub user ilganeli opened a pull request: https://github.com/apache/spark/pull/4708 [SPARK-4655] Split Stage into ShuffleMapStage and ResultStage subclasses Hi all - this patch includes splitting up Stage into ShuffleMapStage and ResultStage and updating their usage within

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75327500 [Test build #27792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27792/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75327654 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75341191 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75353758 [Test build #27805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27805/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75343825 Is this another of the spurious test failure? Seems to be some bizarre thing within ML Lib. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75355985 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75355984 [Test build #27805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27805/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75341185 [Test build #27793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27793/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75284304 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75284266 [Test build #27780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27780/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75284298 [Test build #27780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27780/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75327639 I've submitted a new PR here: https://github.com/apache/spark/pull/4708 --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4708#issuecomment-75327652 [Test build #27792 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27792/consoleFull) for PR 4708 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75286796 [Test build #27782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27782/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75286815 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75288560 [Test build #27783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27783/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75288542 [Test build #27783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27783/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75286811 [Test build #27782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27782/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75288566 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75304156 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75304153 [Test build #27787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27787/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75304143 [Test build #27787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27787/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75306668 [Test build #27788 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27788/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75307867 What do you think about performing only the Stage split in this PR and deferring the other code movement to a separate one? That would make this much easier to

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread ilganeli
Github user ilganeli commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75308162 Hi @JoshRosen, I'm down for making your life easier. Would you still be interested in the other refactoring that I did as part of this patch? --- If your project is

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-20 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75311331 Hi @ilganeli, Those other refactoring might be welcome, since the some of the mega-functions in DAGScheduler can be pretty hard to read. Let's do those

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-19 Thread ilganeli
GitHub user ilganeli opened a pull request: https://github.com/apache/spark/pull/4703 [SPARK-4655] Split Stage into ShuffleMapStage and ResultStage subclasses Hi all - this patch includes two main efforts: 1) I've split up Stage into ShuffleMapStage and ResultStage and

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75201638 [Test build #27771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27771/consoleFull) for PR 4703 at commit

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75201642 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4655] Split Stage into ShuffleMapStage ...

2015-02-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4703#issuecomment-75201629 [Test build #27771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27771/consoleFull) for PR 4703 at commit