Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/6652
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110442688
[Test build #34519 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34519/consoleFull)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110442460
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110442439
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110488256
lgtm too. I think you could go ahead and delete the extra `case _`, though
maybe it makes sense to defer changing `Dependency` to `sealed` to a separate
pr.
---
If
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110476615
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110476551
[Test build #34519 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34519/console)
for PR 6652 at commit
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r32055013
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1384,17 +1400,32 @@ class DAGScheduler(
if (rddPrefs.nonEmpty) {
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110514044
Thanks @squito for taking a look. Yeah I think it makes to change
Dependency to a sealed class but we can do it in a follow up PR. BTW in the
specific case of the
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110566851
@shivaram @kayousterhout this approach addresses my concerns. Thanks for
updating!
---
If your project is set up for it, you can reply to this email and have your
reply
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110121649
@shivaram and I discussed this offline and whether it was possible to
incorporate @sryza's point about the fact that it only makes sense to do this
when there's a
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110185862
[Test build #34475 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34475/consoleFull)
for PR 6652 at commit
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975742
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +137,23 @@ class DAGScheduler(
private[scheduler] val
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975752
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,54 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975747
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
---
@@ -205,4 +205,36 @@ class MapOutputTrackerSuite extends SparkFunSuite {
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975734
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,54 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975212
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +137,23 @@ class DAGScheduler(
private[scheduler] val
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975191
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +137,23 @@ class DAGScheduler(
private[scheduler] val
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975749
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -800,6 +800,50 @@ class DAGSchedulerSuite
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975738
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +137,23 @@ class DAGScheduler(
private[scheduler] val
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110187889
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110187900
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110182683
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110188129
[Test build #34480 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34480/consoleFull)
for PR 6652 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110183130
[Test build #34474 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34474/consoleFull)
for PR 6652 at commit
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975311
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
---
@@ -205,4 +205,36 @@ class MapOutputTrackerSuite extends SparkFunSuite {
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110187549
[Test build #34475 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34475/console)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110187558
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975439
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,54 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975382
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
@@ -800,6 +800,50 @@ class DAGSchedulerSuite
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110199775
[Test build #34474 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34474/console)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110199784
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110182738
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110182980
Thanks @kayousterhout for taking a look and proposing this new design. I
agree that its more cleaner than the top-5 locations and we only get up to 5
locations at the
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31975151
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,54 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110185269
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110185260
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110187193
Mostly LGTM -- just a few readability comments to help whatever future poor
soul wants to update the DAGScheduler code.
@sryza does this new approach
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110204932
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-110204887
[Test build #34480 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34480/console)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109621727
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109622087
[Test build #34374 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34374/console)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109622088
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109640312
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109640273
[Test build #34375 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34375/console)
for PR 6652 at commit
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109621141
@kayousterhout I updated the flag name to be a bit more specific. Let me
know if you have any further comments
---
If your project is set up for it, you can reply to
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109622021
[Test build #34374 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34374/consoleFull)
for PR 6652 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109624541
[Test build #34375 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34375/consoleFull)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109621666
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109623381
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109623306
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31837206
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,44 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109397104
When there is no skew, are there situations where this would lead to worse
performance? E.g. will it make tasks bunch up on nodes more than before and /
or result in
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109390385
@shivaram presumeably you're working on locality for reducers because it
improves overall job time by decreasing the amount of shuffle bytes sent over
the network. Do
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31840291
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,44 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109398051
Also, is there an intuitive justification for why 5 is a good number? It
seems a little weird to me that it's independent of the number of tasks, the
cluster size, and
Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109386418
Ok LGTM other than the small readability suggestion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109439594
@ash211 Yeah, so there are two sets of experiments that I've done which
show benefits with this.
1. The first is in the [OSDI 2014
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109418206
When there is no skew, are there situations where this would lead to
worse performance? E.g. will it make tasks bunch up on nodes more than before
and / or result in
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109450997
Ahhh, I misunderstood and thought that all the reduce tasks for a stage got
the same locality preferences. I withdraw my concern.
---
If your project is set up for it,
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109445990
On the other side, I can also envision situations where it would be really
helpful to request a higher number of preferred locations. Consider a large
YARN cluster where
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109449637
@sryza I'm not sure why the same 5 nodes should get chosen for all the
reduce tasks ? Are you concerned this will happen because we use stable sort ?
We could
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109444936
My worry is the following situation:
* Map outputs from stage 1 exhibit little skew and are distributed evenly
across nodes.
* 5 nodes are chosen as preferred for
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31858018
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,44 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109469337
I briefly chatted with @kayousterhout offline as well and I added a feature
flag for emergencies but kept it on by default.
@JoshRosen @sryza Will be great if
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109490455
[Test build #34329 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34329/console)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109490460
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109469212
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109469219
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109469503
[Test build #34329 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34329/consoleFull)
for PR 6652 at commit
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31764493
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -232,6 +232,12 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31765446
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -232,6 +232,12 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109121081
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31782198
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,41 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109121314
[Test build #34239 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34239/consoleFull)
for PR 6652 at commit
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31782208
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,41 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109111845
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109111833
[Test build #34222 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34222/consoleFull)
for PR 6652 at commit
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31782160
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
---
@@ -205,4 +205,29 @@ class MapOutputTrackerSuite extends SparkFunSuite {
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31782152
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +137,15 @@ class DAGScheduler(
private[scheduler] val
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109121038
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109153334
[Test build #34239 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34239/consoleFull)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109153339
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31762789
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +138,14 @@ class DAGScheduler(
private[scheduler] val
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31765038
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1395,6 +1404,23 @@ class DAGScheduler(
return locs
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109085540
[Test build #34220 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34220/consoleFull)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109091021
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31766803
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,32 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31777183
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,41 @@ private[spark] class MapOutputTrackerMaster(conf:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109086423
[Test build #34220 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34220/consoleFull)
for PR 6652 at commit
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31775435
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,32 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109086435
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31775420
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,32 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31764399
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -21,7 +21,7 @@ import java.io._
import
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31765116
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -137,6 +138,14 @@ class DAGScheduler(
private[scheduler] val
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109069862
[Test build #34210 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34210/consoleFull)
for PR 6652 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/6652#issuecomment-109069872
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31777838
--- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ---
@@ -284,6 +291,41 @@ private[spark] class MapOutputTrackerMaster(conf:
SparkConf)
Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31777847
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala
---
@@ -205,4 +205,29 @@ class MapOutputTrackerSuite extends SparkFunSuite {
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/6652#discussion_r31775183
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1395,6 +1404,23 @@ class DAGScheduler(
return locs
1 - 100 of 137 matches
Mail list logo
