Steve Niemitz created BEAM-9308:
-----------------------------------
Summary: Optimize state cleanup at end-of-window
Key: BEAM-9308
URL: https://issues.apache.org/jira/browse/BEAM-9308
Project: Beam
Issue Type: Improvement
Components: runner-dataflow
Reporter: Steve Niemitz
Assignee: Steve Niemitz
When using state with a large keyspace, you can end up with a large amount of
state cleanup timers set to fire all 1ms after the end of a window. This can
cause a momentary (I've observed 1-3 minute) lag in processing while windmill
and the java harness fire and process these cleanup timers.
By spreading the firing over a short period after the end of the window, we can
decorrelate the firing of the timers and smooth the load out, resulting in much
less impact from state cleanup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
