tgravescs commented on issue #23614: [SPARK-26689][CORE]Support blacklisting bad disk directory and retry in DiskBlockManager URL: https://github.com/apache/spark/pull/23614#issuecomment-459356682 one of the comments above says this is for the Driver, but this is going to affect both executors and driver. I'm a bit on the fence about the change as well but a this point think if spark can be more robust we should do it. Note the configs would need to be documented and I agree with @squito would need to look closer at all the cases. This is in getFile but doesn't handle after you initially get the file but still an improvement. I think we should also make sure to log enough info so someone debugging can easily see things were blacklisted, perhaps print # of disks left after blacklist
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
