Github user HeartSaVioR commented on the issue:
https://github.com/apache/spark/pull/21500
@aalobaidi
One thing you may want to be aware is that in point of executor's view,
executor must load at least 1 version of state in memory regardless of caching
versions. I guess you may get better result if you unload entire cache but
leaving the last version you just committed. Cache miss will occur for one of
three cases `2. committed but batch failed afterwards` but it will happen
rarely and still better than cache miss from two of three cases (2 and 3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
