arunmahadevan commented on issue #23912: [SPARK-21029][SS] StreamingQuery should be stopped when the SparkSession is stopped URL: https://github.com/apache/spark/pull/23912#issuecomment-470232959 That patch leverages shutdown hook to invoke a method to stop the active queries. I am not sure leveraging shutdown hook as a callback mechanism for invoking a method under a different context is the right approach. Also not sure it handles concurrent invocations of stop. To keep it simple, the current patch proposes to handle it only at SparkSession. If we want to handle stop from both SparkContext and SparkSession and concurrency, we probably need to introduce a call back (in SparkSession) that gets invoked from sc.stop and the callback ensures that it stops the query only the first time its invoked. Let me know if it makes sense to explore that.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
