In both SDKs this is an unsafe trigger because it will only fire once for the first window (per key), and any subsequent data on the same key will be dropped. In 2.18, we closed BEAM-3288 with PR https://github.com/apache/beam/pull/9960, which detects these cases and fails early. Probably the fix is to add Repeatedly.forever around your AfterWatermark trigger.
This is noted if you read through https://s.apache.org/finishing-triggers-drop-data but it's not super clear from a user perspective. Maybe we should add a statement like "did you mean to wrap it in Repeatedly.forever?" to the error message, and/or update https://s.apache.org/finishing-triggers-drop-data with clear directions for users. +Kenneth Knowles <[email protected]> On Tue, May 5, 2020 at 5:18 AM Eddy G <[email protected]> wrote: > Hey all! > > Recently been updating Beam pipelines up to 2.19, and the following > trigger which previously worked with 2.15 flawlessly has stopped doing so > and the project doesn't even compile now. > > .apply("15min Window", > Window.<GenericRecord>into(FixedWindows.of(Duration.standardMinutes(15))) > .triggering(AfterWatermark > .pastEndOfWindow()) > .withAllowedLateness(Duration.standardMinutes(60)) > .discardingFiredPanes() > ) > > And will complain with the following error. > > Exception in thread "main" java.lang.IllegalArgumentException: Unsafe > trigger may lose data, see > https://s.apache.org/finishing-triggers-drop-data: > AfterWatermark.pastEndOfWindow() > > Reviewing the changelog I don't see any changes regarding AfterWatermark. > Am I missing something? >
