advancedxy commented on pull request #28128:
URL: https://github.com/apache/spark/pull/28128#issuecomment-653102770
Hi, @vinooganesh and @cloud-fan
I'd like to point out that this pr doesn't fix the memory leaky completely.
Once `SessionState` is touched, it will add two more listeners into the
SparkContext, namely `SQLAppStatusListener` and `ExecutionListenerBus`
It can be reproduced easily as
```
test("SPARK-31354: SparkContext only register one SparkSession
ApplicationEnd listener") {
val conf = new SparkConf()
.setMaster("local")
.setAppName("test-app-SPARK-31354-1")
val context = new SparkContext(conf)
SparkSession
.builder()
.sparkContext(context)
.master("local")
.getOrCreate()
.sessionState // this touches the sessionState
val postFirstCreation = context.listenerBus.listeners.size()
SparkSession.clearActiveSession()
SparkSession.clearDefaultSession()
SparkSession
.builder()
.sparkContext(context)
.master("local")
.getOrCreate()
.sessionState // this touches the sessionState
val postSecondCreation = context.listenerBus.listeners.size()
SparkSession.clearActiveSession()
SparkSession.clearDefaultSession()
assert(postFirstCreation == postSecondCreation)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
