anishshri-db opened a new pull request, #40273: URL: https://github.com/apache/spark/pull/40273
### What changes were proposed in this pull request? We have seen some cases where the task exits as cancelled/failed which triggers the abort in the task completion listener for HDFSStateStoreProvider. As part of this, we cancel the backing stream and close the compressed stream. However, different stores such as Azure blob store could throw exceptions which are not caught in the current path, leading to job failures. This change proposes to fix this issue by catching all non fatal exceptions thrown by cancel/close. ### Why are the changes needed? Changes are required to avoid job failures due to exceptions thrown by output stream handlers on abort with the HDFSStateStoreProvider. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Modified a test and simulated a NPE in the abort path and verified that the task and thereby the job fails before this change. After the change, the test passes fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
