vinooganesh commented on pull request #28128: URL: https://github.com/apache/spark/pull/28128#issuecomment-625569183
Hey @cloud-fan - That was my first approach - but the problem then gets back to keeping track of the object creation/destruction of a `SparkSession`. We can add a JVM wide SparkSession listener upon the creation of the first `SparkSession` that we just never remove (even if there are no SparkSessions left), but that feels a bit weird to me. IMHO, A spun up SparkSession should be able to manage and clean up after itself once it's gone. I do agree that the listener itself feels weird to have, especially when it is created multiple time (each time a new `SparkSession` instance is created). To be academically correct, I think on Application end we should also clean up all of the existing SparkSession instances too - but there isn't an easy way to do that without reference counting, keeping a list of active SparkSessions globally, or something similar, which was my first approach. I figure there are a few options - 1) we either modify the listener to truly clean up properly at the end of the application (meaning it would clean up the existing `SparkSeesion`s) 2) We register the listener once and simply just accept that after the first `SparkSession` is created, that listener will be attached to the context in perpetuity 3) We create lifecycle methods (like what's currently in this PR) that allow a session to clean up after itself. My preferred approach is #3, given that it seems to be the most "correct", but I know that's subjective. Definitely open to your thoughts here. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
