Vinoo Ganesh created SPARK-27337:
------------------------------------
Summary: QueryExecutionListener never cleans up listeners from the
bus after SparkSession is closed
Key: SPARK-27337
URL: https://issues.apache.org/jira/browse/SPARK-27337
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0, 3.0.0
Reporter: Vinoo Ganesh
Attachments: image001-1.png
As a result of
[https://github.com/apache/spark/commit/9690eba16efe6d25261934d8b73a221972b684f3],
it looks like there is a memory leak (specifically
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131).]
Because the Listener Bus on the context still has a reference to the listener
(even after the SparkSession is cleared), they are never cleaned up. This means
that if you close and remake spark sessions fairly frequently, you're leaking
every single time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
