[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-24 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142831032 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8844 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-24 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-143113277 Alright, merged this, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142703664 Yep, sorry for overlooking that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142704410 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142704459 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142703433 Alright, I made the suggested changes. I don't think we need to make those classes `private[spark]` because they are in `src/test`, right? --- If your project is set up

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142705877 [Test build #42917 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42917/consoleFull) for PR 8844 at commit

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142746043 [Test build #42917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42917/console) for PR 8844 at commit

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142746146 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142746148 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-142802670 Alright, let me know if you guys have any other comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-22 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40170084 --- Diff: core/src/test/scala/org/apache/spark/scheduler/CustomShuffledRDD.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40043438 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -323,6 +351,30 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf)

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r39988247 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -323,6 +351,30 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf)

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40027112 --- Diff: core/src/test/scala/org/apache/spark/scheduler/CustomShuffledRDD.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40025584 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -134,11 +134,24 @@ private[spark] abstract class MapOutputTracker(conf:

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40026880 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -323,6 +351,30 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf)

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-21 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r40027092 --- Diff: core/src/test/scala/org/apache/spark/scheduler/CustomShuffledRDD.scala --- @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141841907 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141841910 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141842178 [Test build #42739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42739/consoleFull) for PR 8844 at commit

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141852573 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141852539 [Test build #42739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42739/console) for PR 8844 at commit

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141852575 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread mateiz
GitHub user mateiz opened a pull request: https://github.com/apache/spark/pull/8844 [SPARK-9852] Let reduce tasks fetch multiple map output partitions This makes two changes: - Allow reduce tasks to fetch multiple map output partitions -- this is a pretty small change to

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/8844#issuecomment-141841616 @shivaram, @JoshRosen, @zsxwing this may be relevant to you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-9852] Let reduce tasks fetch multiple m...

2015-09-20 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/8844#discussion_r39935722 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -474,9 +495,9 @@ class DAGSchedulerSuite test("run