squito commented on a change in pull request #24375: [SPARK-27474][CORE] avoid
retrying a task failed with CommitDeniedException many times
URL: https://github.com/apache/spark/pull/24375#discussion_r276787245
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -1389,6 +1389,14 @@ private[spark] class DAGScheduler(
event.reason match {
case Success =>
+ // An earlier attempt of a stage (which is zombie) may still have
running tasks. If these
+ // tasks complete, they still count and we can mark the corresponding
partitions as
+ // finished. Here we notify the task scheduler to skip running tasks
for the same partition,
+ // to save resource.
+ if (task.stageAttemptId < stage.latestInfo.attemptNumber()) {
+ taskScheduler.notifyPartitionCompletion(stageId, task.partitionId)
Review comment:
you could pass in `event.taskInfo` here, or perhaps even jsut
`event.taskInfo.duration`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
