Ngone51 commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-656148667
@srowen Let me help explain a bit more: `activeThreadSession` is a `InheritableThreadLocal`. Therefore, any child thread of the Spark main thread could keep a reference to the `SparkSession` (implemented by `childValue()` by default). In this case, `org.apache.hive.common.util.ShutdownHookManager` creates the hook thread, which should be considered as a child thread of the Spark main thread, during its initialization and adds the hook thread to `java.lang.ApplicationShutdownHooks` at the end. Thus, when we stop the `SparkSession`, its reference is still kept at `java.lang.ApplicationShutdownHooks` since JVM hasn't shut down. The new idea is to override the function `childValue()` to pass the custom value to the child threads. For child threads which created by Spark, we pass the active `SparkSession`. Otherwise, we pass `null` so that those child threads won't keep references of `SparkSession`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
