vinooganesh commented on issue #24807: [SPARK-27958][SQL] Stopping a SparkSession should not always stop Spark Context URL: https://github.com/apache/spark/pull/24807#issuecomment-500871806 Hey @srowen - I'm not super thrilled with using reference counting here either, but I can't think of an easier way to fix the inconsistency here. Definitely open to ideas. There is a memory leak here (where even after the destruction of a Spark Session, the listener added onto the context by the session is never cleaned up, but I believe this be remedied by https://github.com/apache/spark/pull/24807/files#diff-d91c284798f1c98bf03a31855e26d71cR967). The main thing I'm trying to address is what seems like a confusing relationship between a `SparkSession` and a `SparkContext`. Intuitively and per the initial blog post about `SparkSession`, I would that a session is like a networking session - a short lived engagement that is used, and then destroyed. In the current model, the destruction of the session destroys the context as well, which seems like incorrect behavior. A more concrete example of this occurs when using temp tables. Let's say a user in session A creates a temp table, and a user in session B is working on other things. There are 2 guarantees that I believe should be made - 1) The user in session B cannot read the temp table in session A, and 2) the destruction of session A should not harm the user in session B. It seems that in this case, the latter invariant is being broken. Does that help clear things up? Apologies if they were unclear from the thread.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
