otterc commented on code in PR #36293:
URL: https://github.com/apache/spark/pull/36293#discussion_r855600640
##########
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##########
@@ -3709,8 +3709,8 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
assertDataStructuresEmpty()
}
- test("SPARK-32920: Merge results should be unregistered if the running stage
is cancelled" +
- " before shuffle merge is finalized") {
+ test("SPARK-32920: Merge results should not be unregistered if a determinate
running stage is " +
+ "cancelled before shuffle merge is finalized") {
Review Comment:
> we only want the merge results not to be unregistered in the case of stage
retry but for cancelled stage it is fine to remove it right?
We are unable to differentiate between a cancelled stage or a stage that
will be re-attempted in `handleShuffleMergeFinalized`. So, for a determinate
stage we don't unregister the merge results.
I will rephrase the test name
##########
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##########
@@ -3709,8 +3709,8 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSparkContext with Ti
assertDataStructuresEmpty()
}
- test("SPARK-32920: Merge results should be unregistered if the running stage
is cancelled" +
- " before shuffle merge is finalized") {
+ test("SPARK-32920: Merge results should not be unregistered if a determinate
running stage is " +
+ "cancelled before shuffle merge is finalized") {
Review Comment:
> we only want the merge results not to be unregistered in the case of stage
retry but for cancelled stage it is fine to remove it right?
We are unable to differentiate between a cancelled stage or a stage that
will be re-attempted in `handleShuffleMergeFinalized`. So, for a determinate
stage we don't unregister the merge results.
I will rephrase the test name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
