Ngone51 commented on a change in pull request #31839:
URL: https://github.com/apache/spark/pull/31839#discussion_r594839204
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala
##########
@@ -76,7 +76,11 @@ trait QueryExecutionListener {
class ExecutionListenerManager private[sql](session: SparkSession,
loadExtensions: Boolean)
extends Logging {
- private val listenerBus = new ExecutionListenerBus(session)
+ private val listenerBus = new ExecutionListenerBus(session.sessionUUID)
+ session.sparkContext.listenerBus.addToSharedQueue(listenerBus)
+ session.sparkContext.cleaner.foreach { cleaner =>
Review comment:
To be honest, no. It fixes the memory leak so we generally always want
it takes effect. And the `ContextCleaner` is an experienced framework that we
can leverage to fix this issue elegantly without aggressive changes.
The config is introduced since Spark 1.0.0 and enabled by default. So I
personally think it's acceptable to assume users run applications with cleaner
enabled in most cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
