yaooqinn commented on issue #25941: [WIP][SPARK-29257][Core][Shuffle] Use task attempt number as noop reduce id to handle disk failures during shuffle URL: https://github.com/apache/spark/pull/25941#issuecomment-535508591 I prefer we propagate attemptNumber with map statuses The writer can produce “shuffle_$shuffleId_$mapId_$attemptNumber_0.index(data)”. For both external and internal shuffle services, it can use (shuffleId, mapId, attemptNumber) to identify them. Besides, in most cases attemptNumber is default to 0, we can only track those non-zero ones.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
