I think the best answer is "yes" we should fire all timers before exit.
This is the subject of https://issues.apache.org/jira/browse/BEAM-2535 which is a fairly significant enhancement to the model. In this proposal, every timer is treated like an input with a timestamp and that is independent of the specification of when to deliver the input. Right now, processing time timers have no event time timestamp associated with them, nor any watermark hold. So the window expires and they are dropped as late data eventually. This is correct according to the current situation, but we should change it. However, I don't think a pipeline should necessarily actually wait in processing time. One of the main uses of the unified batch/streaming model is to do historical re-processing using the same logic that you used for real-time processing. So in a historical "batch" query, you want all the same callbacks, but you should call them as fast as possible. Semantically, it is the same as a fast clock / slow computation anyhow. Kenn On Fri, Jul 21, 2017 at 6:38 AM, Shen Li <cs.she...@gmail.com> wrote: > If max watermarks arrive at all transforms before some processing time > timers fire, should the Pipeline wait till all timers fire before turning > to DONE state? > > Thanks, > Shen >
