I filed https://github.com/apache/beam/pull/11622 to add a "did you mean" to the error message. Also updated the tl;dr in https://s.apache.org/finishing-triggers-drop-data so people using currently released SDKs will get to the fix sooner when they visit that link:
TL;DR: Almost all uses of triggers that "finish" are data loss bugs. These triggers have been disabled since Beam 2.18. Most Beam users will want to wrap affected triggers in Repeatedly.forever(...). On Tue, May 5, 2020 at 9:44 AM Kyle Weaver <[email protected]> wrote: > > Maybe we should add a statement like "did you mean to wrap it in > Repeatedly.forever?" to the error message > > +1. IMO the less indirection between the user and the fix, the better. > > On Tue, May 5, 2020 at 12:08 PM Luke Cwik <[email protected]> wrote: > >> Pointing users to the website with additional details in the error >> messages would likely help as well. >> >> On Tue, May 5, 2020 at 8:45 AM Brian Hulette <[email protected]> wrote: >> >>> In both SDKs this is an unsafe trigger because it will only fire once >>> for the first window (per key), and any subsequent data on the same key >>> will be dropped. In 2.18, we closed BEAM-3288 with PR >>> https://github.com/apache/beam/pull/9960, which detects these cases and >>> fails early. Probably the fix is to add Repeatedly.forever around your >>> AfterWatermark trigger. >>> >>> This is noted if you read through >>> https://s.apache.org/finishing-triggers-drop-data but it's not super >>> clear from a user perspective. Maybe we should add a statement like "did >>> you mean to wrap it in Repeatedly.forever?" to the error message, and/or >>> update https://s.apache.org/finishing-triggers-drop-data with clear >>> directions for users. +Kenneth Knowles <[email protected]> >>> >>> On Tue, May 5, 2020 at 5:18 AM Eddy G <[email protected]> wrote: >>> >>>> Hey all! >>>> >>>> Recently been updating Beam pipelines up to 2.19, and the following >>>> trigger which previously worked with 2.15 flawlessly has stopped doing so >>>> and the project doesn't even compile now. >>>> >>>> .apply("15min Window", >>>> Window.<GenericRecord>into(FixedWindows.of(Duration.standardMinutes(15))) >>>> .triggering(AfterWatermark >>>> .pastEndOfWindow()) >>>> .withAllowedLateness(Duration.standardMinutes(60)) >>>> .discardingFiredPanes() >>>> ) >>>> >>>> And will complain with the following error. >>>> >>>> Exception in thread "main" java.lang.IllegalArgumentException: Unsafe >>>> trigger may lose data, see >>>> https://s.apache.org/finishing-triggers-drop-data: >>>> AfterWatermark.pastEndOfWindow() >>>> >>>> Reviewing the changelog I don't see any changes regarding >>>> AfterWatermark. Am I missing something? >>>> >>>
