xuanyuanking commented on issue #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24110#issuecomment-476212889 Copy that, I'll try in this way in the following commit. ``` My initial idea is to extend the shuffle id with a generation number, and shuffle id becomes a (int, int) instead of a single int. ``` I met some problem in this way because while the shuffleId is created in shuffle dependency, there's no generation number. Maybe we still need to keep the generation number in ShuffleStatus.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
