Sam Whittle created BEAM-3776:
---------------------------------
Summary: StateMerging.mergeWatermarks sets a late watermark hold
for late merging windows that depend only on the window
Key: BEAM-3776
URL: https://issues.apache.org/jira/browse/BEAM-3776
Project: Beam
Issue Type: Bug
Components: runner-core
Affects Versions: 2.3.0, 2.2.0, 2.1.0
Reporter: Sam Whittle
Assignee: Sam Whittle
WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take to
not add holds that would be before the input watermark.
However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the
window depends only on window, sets a hold for the end of the window regardless
of the input watermark.
Thus if you have a WindowingStrategy such as:
WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
.withMode(AccumulationMode.DISCARDING_FIRED_PANES)
.withTrigger(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
.withLateFirings(AfterPane.elementCountAtLeast(10))))
.withAllowedLateness(allowedLateness))
and you merge windows that are late, you might end up holding the watermark
until the allowedLateness has passed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
