Hey Sean - Cool, maybe I'm misunderstanding the intent of clearing a session vs. stopping it.
The cause of the leak looks to be because of this line here https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131. The ExecutionListenerBus that's added persists forever on the context's listener bus (the SparkContext ListenerBus has an ExecutionListenerBus). I'm trying to figure out the place that this cleanup should happen. With the current implementation, calling SparkSession.stop will clean up the ExecutionListenerBus (since the context itself is stopped), but it's unclear to me why terminating one session should terminate the JVM-global context. Possible my mental model is off here, but I would expect stopping a session to remove all traces of that session, while keeping the context alive, and stopping a context would, well, stop the context. If stopping the session is expected to stop the context, what's the intended usage of clearing the active / default session? Vinoo On 4/2/19, 10:57, "Sean Owen" <sro...@gmail.com> wrote: What are you expecting there ... that sounds correct? something else needs to be closed? On Tue, Apr 2, 2019 at 9:45 AM Vinoo Ganesh <vgan...@palantir.com> wrote: > > Hi All - > > I’ve been digging into the code and looking into what appears to be a memory leak (https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D27337&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=JUsN7EzGimus0jYxyj47_xHYUDC6KnxieeUBfUKTefk&e=) and have noticed something kind of peculiar about the way closing a SparkSession is handled. Despite being marked as Closeable, closing/stopping a SparkSession simply stops the SparkContext. This changed happened as a result of one of the PRs addressing https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D15073&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=Nd9eBDH-FDdzEn_BVt2nZaNQn6fXA8EfVq5rKGztOUo&e= in https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_12873_files-23diff-2Dd91c284798f1c98bf03a31855e26d71cR596&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=RM9LrT3Yp2mf1BcbBf1o_m3bcNZdOjznrogBLzUzgeE&e=. > > > > I’m trying to understand why this is the intended behavior – anyone have any knowledge of why this is the case? > > > > Thanks, > > Vinoo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
