HeartSaVioR commented on PR #37935: URL: https://github.com/apache/spark/pull/37935#issuecomment-1255769799
I gave a feedback offline but also duplicate here for the history. `getStateStoreProvider` also report active provider instance and get “inactive provider instances” which could be unloaded without maintenance. We can defer these instances to be unloaded till next maintenance interval, but this may not be something we want, since we are effectively reverting SPARK-33827. Technically, what we really want to do is just to ensure all state store provider instances run maintenance "at least once". That said, ideally we should not perform maintenance on unloading of provider if it has run the maintenance at least once. In `getStateStoreProvider`, we should only defer some of inactive provider instances which didn't run any maintenance yet. In maintenance task, we need to let provider be unloaded without maintenance if it ran maintenance at least once. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
