Hi Tim, Thanks for the detailed explanation. I understand that the sequence would be beginWindow (x) -> endWindow (x) -> checkpointed (x) -> beginWindow (x+1)
However when I try to print out the window ids in beginWindow, endWindow and checkpointed calls, I see x and x-1 respectively. I.e. If the window just before checkpoint is 100, I see that the checkpointed call had window id 99. Note: This is observed in the local mode of Apex. Thanks -Bhupesh On 10-Nov-2015 11:25 pm, "Timothy Farkas" <[email protected]> wrote: > Hi Bhupesh, > > The sequencing of checkpoint called in relation to beginWindow and > endWindow depends on how your APPLICATION_WINDOW_COUNT and > CHECKPOINT_WINDOW_COUNT are configured. If the two WINDOW_COUNTs are not > configured to be the same then it is possible that checkpointed is called > within an application window. So the sequence of events would be this: > > beginWindow -> checkpointed -> endWindow > > If the APPLICATION_WINDOW_COUNT and CHECKPOINT_WINDOW_COUNT are the same > (default). Then the sequence of calls would be this: > > beginWindow (100) -> endWindow (100) -> checkpointed (100) -> beginWindow > (101) > > The numbers in the sequence represent possible streaming window Ids that > each call would be associated with. > > The StateMachine which calls these callbacks for non-input operators is in > GenericNode.java. > > Tim > > On Tue, Nov 10, 2015 at 3:36 AM, Bhupesh Chawda <[email protected]> > wrote: > > > Hi Chetan / Community, > > > > Can someone please elaborate on why the window id supplied to > > CheckpointListener and the Operator would differ. > > I tried looking at the window ids of checkpointed() and the beginWindow() > > calls and they differ by 1. Don't know why this should be the case. > > > > Thanks. > > -Bhupesh > > > > On Thu, Sep 17, 2015 at 5:56 AM, Chetan Narsude <[email protected]> > > wrote: > > > > > Short answer is yes. > > > > > > All the control tuples are scheduled to be delivered outside of the > > window. > > > As checkpointed callback is triggered because of CHECKPOINT control > > tuple, > > > it will happen after endWindow and before the next beginWindow. > > > > > > The windowId supplied to CheckpointListener and the one provided to > > > Operator need not match even though the sequence is defined. So I am > > > curious how you intend to use this knowledge. > > > > > > -- > > > Chetan > > > > > > > > > On Tue, Sep 15, 2015 at 8:31 AM, Thomas Weise <[email protected]> > > > wrote: > > > > > > > It has not changed the operator execution model. State serialization > is > > > > still synchronous, write to HDFS is taken out of the operator thread. > > > > > > > > On Tue, Sep 15, 2015 at 8:18 AM, Amol Kekre <[email protected]> > > > wrote: > > > > > > > > > > > > > > Sent too soon. Has asynchronous checkpointing changed this? > > > > > > > > > > Amol > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > On Sep 15, 2015, at 12:38 AM, Bhupesh Chawda < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > Hi All, > > > > > > > > > > > > Is it safe to assume that the checkpointed() and the > beginWindow() > > > > calls > > > > > > are sequenced? > > > > > > In other words, are these calls part of the same thread and may > not > > > run > > > > > in > > > > > > parallel? > > > > > > > > > > > > Thanks. > > > > > > > > > > > > -- > > > > > > -Bhupesh > > > > > > > > > > > > > > >
