viirya commented on pull request #30770: URL: https://github.com/apache/spark/pull/30770#issuecomment-747182349
> The problem explanation sounds me as we should unload ASAP whenever possible instead of delaying, right? > > Providing TTL would delay the unload more than current, even giving less luck on encountering problem. We've said "inconsistent" as unload is done between 0 ~ maintenance interval, but TTL doesn't ensure the state will get evicted exactly at that time, hence not sure about the difference. That only draws a line to set lower bound, but to address the problem lower bound should be minimized. The upper bound is between TTL ~ (TTL + maintenance interval) which is higher than current. I'm not sure the context, why we don't do unloading asap. TTL is feasible approach I can think of, to address it, under current maintenance mechanism. Even just set a lower bound, it is still more valuable, if we consider the further change to reuse previous stores more reliably. 0 ~ maintenance is not useful as we cannot make sure if a previous store is still there. TTL at least lets us know it is there until TTL. > To resolve the problem being described, driver should be also able to tell the executor that another executor is registered as active for the state so the executor should be safe to unload and preferably immediate. To do that, bi-directional communication would be required. Ideally, yes, if we can make uploading as exactly as configured in TTL, it is best. Not sure how many change is required. I will try to look at this too. Anyway, consider the further change I will make to reuse previous stores, I think it makes this TTL more useful. WDYT? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
