igorcalabria commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main URL: https://github.com/apache/spark/pull/25785#issuecomment-531524626 > I see, can that thread be a daemon? If System.exit is viable (i.e. immediately stopping daemon threads) then it should be. But if not, then yeah such a thread needs to be shut down cleanly somehow during the shutdown process. This could be a shutdown hook. I don't think there's an option to create a daemon thread in this case. This was already discussed on https://github.com/square/okhttp/issues/3339 I'm sorry, but I didn't understand what you meant about the viability of `System.exit`. There's already a shutdown logic for the kubernetes client https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala#L115 but this only called if the shutdownHook runs or if the user calls `SparkSession#stop()`. Calling system.exit is just a way to ensure that the hooks run after the user's main code. Everything is still cleaned up nicely since the System.exit does not bypass the registered hooks. > Hi, @igorcalabria . > According to the test result, this PR seems to break `Spark Thrift Server` module at least UT level. The following is one example failure. To be considered as mergeable, this PR should pass all UTs at least. Could you focus on fixing that module? > > ``` > $ build/sbt -Phive-thriftserver "hive-thriftserver/test-only *.SingleSessionSuite" > ... > [info] Tests: succeeded 0, failed 3, canceled 0, ignored 0, pending 0 > [info] *** 3 TESTS FAILED *** > [error] Failed: Total 3, Failed 3, Errors 0, Passed 0 > [error] Failed tests: > [error] org.apache.spark.sql.hive.thriftserver.SingleSessionSuite > [error] (hive-thriftserver/test:testOnly) sbt.TestsFailedException: Tests unsuccessful > [error] Total time: 258 s, completed Sep 14, 2019 4:16:09 PM > ``` @dongjoon-hyun I'll take a look.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
