Hey Reza,
Indeed, the batch and streaming behaviors differ slightly, though they are both
compliant with the Beam model.
Glad you could correct the timer. Let us know how it goes with the Flink Runner.
Thanks,
Max
On 16.12.18 02:52, Reza Ardeshir Rokni wrote:
Hya Max,
Thank you for the reply and I realized I had not given enough background context
on where the timers are set. So a little more color...
1- Apply Fixed Window of X to stream
2- Combiner generates aggregators per key.
3- Apply Global Window to aggregators. ( It is understood , that order of
aggregators is not guaranteed in the Global Window)
4- A 'looping timer' is created when a key is first seen. If keyed-state is null
a timer is set to the next aggregators lower window boundary.
...
There is actually a lot more to the pipeline than just those steps as it deals
with the out of order problem and other subtle fun bits...
However your answer did give me the logic mistake I had made.. In step 4 I need
to actually set the timer to the upper boundary as I want it to only fire after
watermark has moved to the end of my fixed window. In batch mode testing this
was not an issue as I understand it the timers all fire after the @Process has
been completed. But in stream mode my mistake showed up. Fun stuff..
I do need to run all the tests with the Flink runner as well, will report back
on how it goes.. :-)
Cheers
Reza
On Fri, 14 Dec 2018 at 21:15, Maximilian Michels <[email protected]
<mailto:[email protected]>> wrote:
Hi Reza,
You are asking about the order in which @OnTimer and @ProcessElement are
called.
I'm not sure about the Dataflow Runner, but in the Flink Runner there is no
strict order between the two, other than the guarantees that apply to window
processing and readiness of timers.
To be able to set the timer in "lower bound" of the window (the minimum
timestamp), you will have to process an element which registers the timer.
So
you can't guarantee the timer fires beforehand. If you meant the "upper
bound"
(the maximum timestamp), then it can be guaranteed that the timer fires last
because the timer will fire when the watermark is moved to or past the
maximum
timestamp.
Generally, elements will be processed as they arrive in the window. Timers
are
fired when they are ready. It is best not to make assumptions based on when
elements arrive which belong to the same window. However, you can be sure
that
timers fire after they become eligible.
Thanks,
Max
On 14.12.18 10:43, Reza Ardeshir Rokni wrote:
> Hi,
>
> I believe a bug in my timeseries code is because of something I missed in
the
> sequence of OnTimer / ProcessElement when in stream mode.
>
> If a timer has been set at the lower boundary of a window and elements
arrive in
> that windows keyed state, which will fire first? The @OnTimer or
@ProcessElement ?
>
> Cheers
> Reza
