Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/4055#discussion_r33965943
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1193,8 +1193,10 @@ class DAGScheduler(
// TODO: This will be really slow if we keep accumulating shuffle
map stages
for ((shuffleId, stage) <- shuffleToMapStage) {
stage.removeOutputsOnExecutor(execId)
- val locs = stage.outputLocs.map(list => if (list.isEmpty) null
else list.head).toArray
- mapOutputTracker.registerMapOutputs(shuffleId, locs, changeEpoch
= true)
+ if (!runningStages.contains(stage)) {
--- End diff --
@suyanNone I'm not entirely sure I understand -- let me try to explain it
back to you, tell me if this is correct:
this change is not for **correctness**, its more for efficiency. There is
no point in registering map output for running stages, since we'll only
register some subset of the stages full output. Eventually when the stage
completes, the full set of map outputs will be registered, and that is all that
matters anyway. So there is no point registering the map output here.
Does that sound right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
