viirya edited a comment on pull request #30770: URL: https://github.com/apache/spark/pull/30770#issuecomment-747325973
I see. It is much clearer now. This is already over the original scope of this PR. The purpose of this change is pretty simple: making store unloading more consistent. The problematic point I saw is, we unload stores arbitrarily, it could be early as immediately, it could be late as maintenance interval. As a complement to current maintenance task, it is not proposed to as a total solution for managing inactive stores. If we are considering better solution for inactive store management than current maintenance task, then yes, TTL approach is far from good. I originally thought the design of maintenance task is to intentionally keep inactive store and make it possible for reuse in later batch if the same provider is scheduled on the same executor. But from on above discussion, I think it is totally wrong. Anyway, I agree that to reduce inactive stores is what we should achieve. I think I will be happy to look into in next if it is achievable. Thanks for the discussion. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
