cloud-fan commented on code in PR #36564:
URL: https://github.com/apache/spark/pull/36564#discussion_r893038044
##########
core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala:
##########
@@ -187,8 +188,8 @@ class OutputCommitCoordinatorSuite extends SparkFunSuite
with BeforeAndAfter {
// The authorized committer now fails, clearing the lock
outputCommitCoordinator.taskCompleted(stage, stageAttempt, partition,
attemptNumber = authorizedCommitter, reason = TaskKilled("test"))
- // A new task should now be allowed to become the authorized committer
- assert(outputCommitCoordinator.canCommit(stage, stageAttempt, partition,
+ // A new task should not be allowed to become stage failed because of may
cause data duplication
Review Comment:
```suggestion
// A new task should not be allowed to become stage failed because of
potential data duplication
```
##########
core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala:
##########
@@ -235,7 +236,8 @@ class OutputCommitCoordinatorSuite extends SparkFunSuite
with BeforeAndAfter {
assert(!outputCommitCoordinator.canCommit(stage, 3, partition,
taskAttempt))
outputCommitCoordinator.taskCompleted(stage, 1, partition, taskAttempt,
ExecutorLostFailure("0", exitCausedByApp = true, None))
- assert(outputCommitCoordinator.canCommit(stage, 4, partition, taskAttempt))
+ // A new task should not be allowed to become stage failed because of may
cause data duplication
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
