Vinoo Ganesh created SPARK-27337: ------------------------------------ Summary: QueryExecutionListener never cleans up listeners from the bus after SparkSession is closed Key: SPARK-27337 URL: https://issues.apache.org/jira/browse/SPARK-27337 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.4.0, 3.0.0 Reporter: Vinoo Ganesh Attachments: image001-1.png
As a result of [https://github.com/apache/spark/commit/9690eba16efe6d25261934d8b73a221972b684f3], it looks like there is a memory leak (specifically [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131).] Because the Listener Bus on the context still has a reference to the listener (even after the SparkSession is cleared), they are never cleaned up. This means that if you close and remake spark sessions fairly frequently, you're leaking every single time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org