attilapiros edited a comment on pull request #31742: URL: https://github.com/apache/spark/pull/31742#issuecomment-796003983
> When Spark is shut down, all references go out of scope. These changes bring Spark's behavior in line with the intended (and documented) behavior. I think you are wrong here: checkpointing should outlive the unexpected shutdowns. So there is a very important difference between the reference goes out of scope during a normal execution (in this case cleanup is expected depending on the config you mentioned) and when a references goes out of scope because of an unexpected error (in this case you should keep the checkpoint data). So even after an unexpected exit the next run of the same app could pick up the checkpointed data. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
