dongjoon-hyun commented on PR #42301: URL: https://github.com/apache/spark/pull/42301#issuecomment-3559346468
Thank you for updating your PR, @Kimahriman . To @pan3793 , IIRC, you are using [YARN in a way](https://github.com/apache/spark/pull/52923#discussion_r2501639415). I'm wondering how you handle this RocksDB issue (in this PR description)? > The problem with this is that if an executor crashes for some reason (like OOM) and the shutdown hooks don't get run, this directory will stay around forever until the application finishes, which can cause jobs to slowly accumulate more and more temporary space until finally the node manager goes unhealthy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
