Hi users,
** Do you use TimestampCombiner.EARLIEST with SlidingWindows or other
overlapping windows? If no, then you can stop reading now. **
We are considering a change to simplify how timestamps are computed for
aggregations in this case. Warning: this is a bit complicated and a curious
corner of Beam, but quite important if you do use this configuration.
Currently, when you use EARLIEST with overlapping windows like
SlidingWindows, the output timestamp is shifted later, to the beginning of
the next window. This is a hack to allow downstream watermarks to progress.
To illustrate: consider the windows A-B and C-D which overlap, and an
element with timestamp x that is earlier than C:
A-x------------------B
C----------------------D
Today, the aggregation for window A-B will have the timestamp C, not x.
We are considering changing this to simply use the minimum: x. This is more
expected behavior. It will have the possible negative effect of holding the
watermark so that downstream aggregations are delayed.
Kenn
