skonto commented on a change in pull request #24796: [SPARK-27900][CORE] Add
uncaught exception handler to the driver
URL: https://github.com/apache/spark/pull/24796#discussion_r291486421
##########
File path: core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
##########
@@ -204,6 +204,11 @@ private [util] class SparkShutdownHookManager {
hooks.synchronized { hooks.remove(ref) }
}
+ def clear(): Unit = {
Review comment:
@zsxwing interesting... I use the jvm in the alpine docker image used for
Spark on K8s [openjdk:8-alpine
](https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile#L18)
```
sh-4.4$ java -version
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (IcedTea 3.11.0) (Alpine 8.201.08-r1)
OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode
```
which is the latest...
Btw from what I read there is no guarantee shutdown hooks will run till
completed...
https://dzone.com/articles/know-jvm-series-2-shutdown, that is why I thought
just exiting is the most reliable option and that is why I cleared the hooks,
although not satisfying in every case.
"First thing to keep in mind is that it is not guaranteed that shutdown
hooks will always run. If the JVM crashes due to some internal error, then it
might crash down without having a chance to execute a single instruction."
In addition, even if this will not happen often, shutdown hooks need to
finish fast anyway so you stand a chance to finish (especially with jvm errors)
and waiting in a join when you dont know the state of things is dangerous
anyway.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
