vanzin commented on issue #23563: [SPARK-26634]Do not allow task of FetchFailureStage commit in OutputCommitCoordinator URL: https://github.com/apache/spark/pull/23563#issuecomment-463671990 > the partition 1 of stage 0 is marked as completed, why the scheduler need to schedule this task? Because it completes after the next stage attempt starts, so the scheduler creates new tasks for all the tasks that did not finish in the previous attempt up to that point. Anyway, if Imran's fix covers this case, that's good, no need to make this more complicated.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
