HeartSaVioR commented on pull request #30770: URL: https://github.com/apache/spark/pull/30770#issuecomment-747178340
The problem explanation sounds me as we should unload ASAP whenever possible instead of delaying, right? Providing TTL would delay the unload more than current, even giving less luck on encountering problem. We've said "inconsistent" as unload is done between 0 ~ maintenance interval, but TTL doesn't ensure the state will get evicted exactly at that time, hence not sure about the difference. That only draws a line to set lower bound, but to address the problem lower bound should be minimized. The upper bound is between lower bound ~ (TTL + maintenance interval) which is higher than current. To resolve the problem being described, driver should be also able to tell the executor that another executor is registered as active for the state so the executor should be safe to unload and preferably immediate. To do that, bi-directional communication would be required. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
