hywang203 commented on pull request #31742: URL: https://github.com/apache/spark/pull/31742#issuecomment-790987483
Why [this ](https://github.com/apache/spark/blob/3a299aa6480ac22501512cd0310d31a441d7dfdc/core/src/main/scala/org/apache/spark/ContextCleaner.scala#L179)didn't work? Can I get more analysis on why current code is broken? Are you saying (purely my guess) that keepCleaning process is interrupted as soon as sc is stopped thus did not get chance to do actual cleaning? In that case, you find something general...checkpoint cleaning is not the only method that need to be moved to shutdown hook then... That would explain why you are able to repro locally but I'm not able to repro with my real script, my script does run much longer after last checkpoint. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
