venkata91 commented on a change in pull request #34122:
URL: https://github.com/apache/spark/pull/34122#discussion_r792273604
##########
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -3686,14 +3686,13 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
completeNextStageWithFetchFailure(3, 0, shuffleDep)
scheduler.resubmitFailedStages()
- // Make sure shuffle merge is disabled for the retry
val stage2 = scheduler.stageIdToStage(2).asInstanceOf[ShuffleMapStage]
- assert(!stage2.shuffleDep.shuffleMergeEnabled)
+ assert(stage2.shuffleDep.shuffleMergeEnabled)
Review comment:
This is due to the change we made
[here](https://github.com/apache/spark/pull/34122/files#diff-85de35b2e85646ed499c545a3be1cd3ffd525a88aae835a9c621f877eebadcb6L1421-L1422)
Earlier if enough mergers are not available, we would set
`shuffleMergeEnabled = false` and `shuffleMergeFinalized` would return true
even though it is not merge finalized as shuffle merge was not enabled in the
first place based on this
[code](https://github.com/apache/spark/blob/f6128a6f4215dc45a19209d799dd9bf98fab6d8a/core/src/main/scala/org/apache/spark/Dependency.scala#L154).
This is actually not quite right which is explained here as part of this
[comment](https://github.com/apache/spark/pull/34122/files#r785245294). Since
the `shuffleMergeFinalized` behavior is changed, in the retry
`shuffleMergeEnabled` will be `true (or mergerLocs.nonEmpty)` as enough mergers
are now available and shuffle merge is not finalized as shuffle was not merge
enabled in the first attempt due to not enough mergers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
