Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21500
@aalobaidi
I just would like to see the benefit of unloading the version of state
which is expected to be read from the next batch. Totally I agree current
mechanism of cache is excessive, but we can still avoid reloading in every
batch. Are you considering multiple stages which executor is encouraged to
clean up memory as much as it can, despite of redundant reloading state?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
