viirya edited a comment on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747135903
> Once the state gets huge, the cost to load from state store should be
huge and this PR helps to remedy the situation, but the cost to maintain the
same store across different executors isn't also small. So that's not a clear
benefit but kind of trade-off.
Yes. It's a trade-off here. We don't have such flexibility for users for
now. Normally we will hope to have good trade-off between loading checkpointed
states and keeping same store across different executors. I think this is why
we don't remove it immediately.
Currently we will face some cases like a store is loaded in a new executor B
right before next maintenance task, so the task removes the previous store in
the executor A. But another store might be removed after the maintenance
interval.
This makes the loading/unloading behavior inconsistent and hard to reason.
Generally I think it is better to keep a consistent behavior here.
This is not a trivial configuration as wrong config value could lead to bad
case. But it is also why this is just an internal configuration like the
maintenance interval (note: I found I forgot to let it be an internal config).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
