Ben Chambers created BEAM-696:
---------------------------------
Summary: Side-Inputs non-deterministic with merging main-input
windows
Key: BEAM-696
URL: https://issues.apache.org/jira/browse/BEAM-696
Project: Beam
Issue Type: Bug
Components: beam-model
Reporter: Ben Chambers
Assignee: Frances Perry
Side-Inputs are non-deterministic for several reasons:
1. Because they depend on triggering of the side-input (this is acceptable
because triggers are by their nature non-deterministic).
2. They depend on the current state of the main-input window in order to lookup
the side-input. This means that with merging
3. Any runner optimizations that affect when the side-input is looked up may
cause problems with either or both of these.
This issue focuses on #2 -- the non-determinism of side-inputs that execute
within a Merging WindowFn.
Possible solution would be to defer running anything that looks up the
side-input until we need to extract an output, and using the main-window at
that point. Specifically, if the main-window is a MergingWindowFn, don't
execute any kind of pre-combine, instead buffer all the inputs and combine
later.
This could still run into some non-determinism if there are triggers
controlling when we extract output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
