squito commented on issue #23614: [SPARK-26689][CORE]Support blacklisting bad disk directory and retry in DiskBlockManager URL: https://github.com/apache/spark/pull/23614#issuecomment-462931736 I'm sorry I haven't been able to look at this closely, but I do want to express my reservations about this. I'm worried about what'll happen with the external shuffle service when something appears to go wrong in the middle of an application, especially if its a temporary issue. I also want to make sure the memory use of the `migratedFile` structure doesn't grow without bound. the number of files in there could be huge. I guess its OK if it only grows after a failure, as you're trading a failure right now from the DiskStore for an OOM later on. anyway I'm not blocking this, just want to make sure this is thought through carefully before its merged.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
