viirya edited a comment on pull request #30770: URL: https://github.com/apache/spark/pull/30770#issuecomment-747232971
> So IMHO the right direction would be either trying our best to unload inactive state ASAP, or considering the replication as the further improvement. Not somewhere in between. Even the latter wouldn't be an improvement if we could enforce Spark to respect the active executor of the state. Regarding "the replication", I read through the previous comments. I'm not pretty sure if I understand your point correctly. Is it any different than reusing the stores of previous batch? Because seems to me, you are against to have previous stores kept in TTL and reuse them, or I misread your comments? But the replication sounds similar to me. Can you elaborate it? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
