cloud-fan commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-521637405 @vanzin I checked the code: https://github.com/apache/spark/blob/1b416a0c77706ba352b72841d8b6ca3f459593fa/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1355-L1365 Spark will ignore late speculative tasks that are completed after the stage is completed. So at least the worst case won't happen: a speculative task could replace the output of another task after the stage has finished (and thus after the next stage started running) But we still have the contradiction: executor side first shuffle write wins, driver side last shuffle write wins. When we run an indeterminate stage, the downstream stages are always fresh (when we rerun an indeterminate stage, the scheduler rolls back all downstream stages). So it doesn't matter which shuffle write wins, as long as the shuffle write is atomic. We'd better fix this contradiction, but it doesn't cause any real problems.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
