cloud-fan commented on issue #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24110#issuecomment-475780104 > All the block id should be extended by adding attempt id, not only the indeterminate ones. That's what I expect. My initial idea is to extend the shuffle id with a generation id, and shuffle id becomes a (int, int) instead of a single int. With the shuffle id change, the shuffle block id also changes. I think it's a much simpler design comparing to doing different things for different deterministic levels.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
