anishshri-db opened a new pull request, #42716:
URL: https://github.com/apache/spark/pull/42716
### What changes were proposed in this pull request?
Avoid uncaught exception from state store maintenance task thread on error
to prevent executor being killed
### Why are the changes needed?
With the current change, the uncaught exception handler was being triggered
on the executor leading to the executor being killed.
```
/**
* The default uncaught exception handler for Spark daemons. It terminates
the whole process for
* any Errors, and also terminates the process for Exceptions when the
exitOnException flag is true.
*
* @param exitOnUncaughtException Whether to exit the process on
UncaughtException.
*/
private[spark] class SparkUncaughtExceptionHandler(val
exitOnUncaughtException: Boolean = true)
```
Ideally we just want to note the exception which will eventually be picked
up by the maintenance task thread and force the reset, instead of causing the
executor JVM process to be killed. This change fixes this issue.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests
Before:
```
[info] Run completed in 2 seconds, 398 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error]
org.apache.spark.sql.execution.streaming.state.StateStoreSuite
```
After:
```
[info] Run completed in 2 seconds, 339 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
