viirya commented on a change in pull request #25498: [SPARK-28699][Core] Fix a
corner case for aborting indeterminate stage
URL: https://github.com/apache/spark/pull/25498#discussion_r315395782
##########
File path:
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -2741,27 +2741,8 @@ class DAGSchedulerSuite extends SparkFunSuite with
LocalSparkContext with TimeLi
FetchFailed(makeBlockManagerId("hostC"), shuffleId2, 0, 0, "ignored"),
null))
- val failedStages = scheduler.failedStages.toSeq
- assert(failedStages.length == 2)
- // Shuffle blocks of "hostC" is lost, so first task of the
`shuffleMapRdd2` needs to retry.
- assert(failedStages.collect {
- case stage: ShuffleMapStage if stage.shuffleDep.shuffleId == shuffleId2
=> stage
- }.head.findMissingPartitions() == Seq(0))
- // The result stage is still waiting for its 2 tasks to complete
- assert(failedStages.collect {
- case stage: ResultStage => stage
- }.head.findMissingPartitions() == Seq(0, 1))
-
- scheduler.resubmitFailedStages()
-
- // The first task of the `shuffleMapRdd2` failed with fetch failure
- runEvent(makeCompletionEvent(
- taskSets(3).tasks(0),
- FetchFailed(makeBlockManagerId("hostA"), shuffleId1, 0, 0, "ignored"),
- null))
-
- // The job should fail because Spark can't rollback the shuffle map stage.
- assert(failure != null && failure.getMessage.contains("Spark cannot
rollback"))
+ assert(failure != null && failure.getMessage
Review comment:
I feel it is good too to leave some comment here, as this change the test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
