So far been dealing with a scenario in Apache Beam where given a certain HTTP
code, I may be preserving the elements to restart in the next iteration.
Been implementing this with inner code, only using a time trigger.
.apply(
"Sample Window",
Window.into(FixedWindows.of(Duration.standardMinutes(1)))
.triggering(AfterProcessingTime
.pastFirstElementInPane()
.plusDelayOf(Duration.standardSeconds(1)))
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes()
)
I was hardcoding my logic to handle the request of, let's say, 200 events. And
also storing in-memory those events in case the request failed.
However, checking the docs I saw combined triggering...
Repeatedly.forever(AfterFirst.of(
AfterPane.elementCountAtLeast(100),
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardMinutes(1))))
So did the same in my case.
.apply(
"Sample Window",
Window.<KV<String,
String>>into(FixedWindows.of(Duration.standardMinutes(1)))
.triggering(
AfterFirst.of(
AfterPane.elementCountAtLeast(200),
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(1))
)
)
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes()
)
So been wondering now...
If triggered by number of elements within the 1 minute timeframe, what happens
with those events? Are they reprocessed again? Should I manually remove them
from the Window?
I'm also talking in the case the 200 elements fail. How can I make them prevail
in the window?