Anish Shrigondekar created SPARK-42668:
------------------------------------------
Summary: Catch exception while trying to close compressed stream
in HDFSStateStoreProvider abort
Key: SPARK-42668
URL: https://issues.apache.org/jira/browse/SPARK-42668
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 3.4.0
Reporter: Anish Shrigondekar
Catch exception while trying to close compressed stream in
HDFSStateStoreProvider abort
We have seen some cases where the task exits as cancelled/failed which triggers
the abort in the task completion listener for HDFSStateStoreProvider. As part
of this, we cancel the backing stream and close the compressed stream. However,
different stores such as Azure blob store could throw exceptions which are not
caught in the current path, leading to job failures. This change proposes to
fix this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
