Hi Aaron,
I don't think we have such fine grained metrics on per operation state
size, but from your description that "YARN kills containers who are
exceeding their memory limits", I think the root cause is not the state
size but related to the memory consumption of the state backend.
My guess is
Hi,
I’m not sure if there is some simple way of doing that (maybe some other
contributors will know more).
There are two potential ideas worth exploring:
- use periodically triggered save points for monitoring? If I remember
correctly save points are never incremental
- use save point input/out
