igorcalabria commented on issue #25785: [SPARK-27812][CORE] Explicit System.exit after job's main URL: https://github.com/apache/spark/pull/25785#issuecomment-531864348 I actually found the root issue that introduced the non-daemon thread in the kubernetes lib https://github.com/fabric8io/kubernetes-client/pull/1301. It is hardcoding the value of `pingInterval` in okhttp's lib. When that value is higher than 0, it creates a non-daemon thread. Spark <= 2.4.0 worked fine because it sets that value to 0 and used a version prior to the linked PR. We're still setting that value to zero https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala#L83 but it's not being respected by kubernetes lib anymore. I totally agree that System.exit is a overkill solution for this problem and could have unforeseen consequences. You guys can close this PR, I'll ping back with a new PR when kubernetes-client accepts my changes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
