squito commented on issue #23647: [SPARK-26712]Support multi directories for executor shuffle info recovery in yarn shuffle serivce URL: https://github.com/apache/spark/pull/23647#issuecomment-458224607 > it will still cause resource waste, for shuffle will always fail on the node, not not mention that there are chances that the node is not blacklisted there will certainly be some resource waste, but we have to balance complexity vs. how often the issue would occur and how bad the simpler behavior would be. If you have a bad disk, you're definitely losing some shuffle data. Furthermore, any other shuffleMapStages would need to know to not write their output to the bad disk also. Blacklisting should kick in here, and if it doesn't, we should figure out why. Yes, there will be some waste till that happens, but I think we can live with that.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
