jto commented on PR #28944:
URL: https://github.com/apache/beam/pull/28944#issuecomment-1836783366
There's currently an issue in Batch mode specifically with timers because of
a race condition in `ExecutableStageDoFnOperator`.
Because `processElement` is async and the actual waiting for bundle being
processed only happens in `finishBundle`, the following issue 2 issues will
happen:
- Timers need to be forcibly triggered at the end of the bundle in batch
mode otherwise they may be lost. This is easily fixed because a similar hack
exists for processing time events.
- A DoFn that sets a timer will execute and its corresponding bundle will
actually finish, however, while the DoFn `processElement` is successfully
executed the actual registration of the timer is not yet completed (everything
is async since the body of the DoFn runs in a different process / thread). The
following events will happen:
1. Bundle finishes,
3. Timers are set and fired,
3. next bundle is started.
Because the registering and firing of timers happens BETWEEN 2 bundles
(after finish, before start) those timers firing will just be lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
