cloud-fan commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-519329319 The basic assumption of the speculative task is that: the task output is deterministic and we can pick the output of any task which completes first. The assumption is broken with interminate shuffle stage. We shouldn't launch speculative task at all for this case, and thus no need to worry about rolling back w/ speculative tasks. I'm curious to see other use cases of always putting task id in shuffle block id. But according to my experience, it's hard to push for a feature that brings consistent perf overhead in low-level modules like shuffle.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
