zsxwing opened a new pull request #27678: [SPARK-30927][SS]StreamingQueryManager should avoid keeping reference to terminated StreamingQuery URL: https://github.com/apache/spark/pull/27678 ### What changes were proposed in this pull request? Right now `StreamingQueryManager` will keep the last terminated query until `resetTerminated` is called. When the last terminated query has lots of states (a large sql plan, cached RDDs, etc.), it will waste these memory unnecessarily. Actually, what `StreamingQueryManager` really needs is just the exception of the last failed query. This PR changes the internal field of `StreamingQueryManager` to remember the last exception instead to save the memory. ### Why are the changes needed? Avoid keeping memory unnecessarily. ### Does this PR introduce any user-facing change? No ### How was this patch tested? This PR doesn't change any public behaviors. The existing tests have covered the touched codes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
