Kenneth Knowles created BEAM-654:
------------------------------------

             Summary: When and how can merging windows "shrink" or "grow"?
                 Key: BEAM-654
                 URL: https://issues.apache.org/jira/browse/BEAM-654
             Project: Beam
          Issue Type: New Feature
          Components: beam-model
            Reporter: Kenneth Knowles
            Assignee: Kenneth Knowles


The primary example of a merging window today is {{Sessions}} by gap duration, 
in which the merged window is the interval enclosure / span of the windows 
being merged.

However, another reasonable abstract use case is a session identified by id 
with an explicit end event. We might consider that the session ends with no gap 
duration after the end event. In this case, the merged window may be smaller 
than the enclosure of the sub-windows. Sometimes this has been referred to as 
"merging shrinks the window".

Perhaps the only requirement is that the merged window contains the timestamps 
of the data therein, but we should document this clearly. The current spec is 
["Does whatever merging is 
necessary"|https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/WindowFn.java#L106]

There are repercussions for triggers, some documented in the [trigger design 
doc|https://s.apache.org/beam-triggers]: With nonzero allowed lateness, 
{{Sessions}} by gap duration can switch a trigger from ON_TIME or LATE behavior 
back to speculative behavior, or cause another ON_TIME firing. Conversely, 
sessions with abrupt termination/shrinking may have that behavior _as well as_ 
an ON_TIME and subsequent LATE firings due only to the merging (this already 
works properly).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to