Checkpoint notification is different from commit notification. Checkpointed is called when the state is preserved. Committed is called when the preserved state is no longer needed. That happens when the checkpointed windowId is processed by all the operators in the application.
The checkpoint windowIds supplied to an operator are guaranteed to be monotonically increasing during operator instances life cycle (as the windowIds themselves are). If there is a failure, a new instance life cycle begins in which case there would be one time rewind on windowId and hence everything that depends on it. ‹ Chetan On 11/10/15, 10:33 AM, "Munagala Ramanath" <[email protected]> wrote: >Chetan's answer provides a good explanation as well as clarifying that >the difference can be more than 1. > >Since checkpointing (i.e. "commit notification" as Thomas refers to >it) is asynchronous, I'm curious >about whether the window ids in the checkpointed call are guaranteed >to be sequential or if they could >be out of order, i.e. can the checkpointed call see window id 101 >before it sees 100 ? > >Ram > >On Tue, Nov 10, 2015 at 10:27 AM, Bhupesh Chawda ><[email protected]> wrote: >> Hi Tim, >> Thanks for the detailed explanation. >> I understand that the sequence would be >> beginWindow (x) -> endWindow (x) -> checkpointed (x) -> beginWindow >> (x+1) >> >> However when I try to print out the window ids in beginWindow, endWindow >> and checkpointed calls, I see x and x-1 respectively. >> I.e. If the window just before checkpoint is 100, I see that the >> checkpointed call had window id 99. >> >> Note: This is observed in the local mode of Apex. >> >> Thanks >> -Bhupesh >> On 10-Nov-2015 11:25 pm, "Timothy Farkas" <[email protected]> wrote: >> >>> Hi Bhupesh, >>> >>> The sequencing of checkpoint called in relation to beginWindow and >>> endWindow depends on how your APPLICATION_WINDOW_COUNT and >>> CHECKPOINT_WINDOW_COUNT are configured. If the two WINDOW_COUNTs are >>>not >>> configured to be the same then it is possible that checkpointed is >>>called >>> within an application window. So the sequence of events would be this: >>> >>> beginWindow -> checkpointed -> endWindow >>> >>> If the APPLICATION_WINDOW_COUNT and CHECKPOINT_WINDOW_COUNT are the >>>same >>> (default). Then the sequence of calls would be this: >>> >>> beginWindow (100) -> endWindow (100) -> checkpointed (100) -> >>>beginWindow >>> (101) >>> >>> The numbers in the sequence represent possible streaming window Ids >>>that >>> each call would be associated with. >>> >>> The StateMachine which calls these callbacks for non-input operators >>>is in >>> GenericNode.java. >>> >>> Tim >>> >>> On Tue, Nov 10, 2015 at 3:36 AM, Bhupesh Chawda >>><[email protected]> >>> wrote: >>> >>> > Hi Chetan / Community, >>> > >>> > Can someone please elaborate on why the window id supplied to >>> > CheckpointListener and the Operator would differ. >>> > I tried looking at the window ids of checkpointed() and the >>>beginWindow() >>> > calls and they differ by 1. Don't know why this should be the case. >>> > >>> > Thanks. >>> > -Bhupesh >>> > >>> > On Thu, Sep 17, 2015 at 5:56 AM, Chetan Narsude >>><[email protected]> >>> > wrote: >>> > >>> > > Short answer is yes. >>> > > >>> > > All the control tuples are scheduled to be delivered outside of the >>> > window. >>> > > As checkpointed callback is triggered because of CHECKPOINT control >>> > tuple, >>> > > it will happen after endWindow and before the next beginWindow. >>> > > >>> > > The windowId supplied to CheckpointListener and the one provided to >>> > > Operator need not match even though the sequence is defined. So I >>>am >>> > > curious how you intend to use this knowledge. >>> > > >>> > > -- >>> > > Chetan >>> > > >>> > > >>> > > On Tue, Sep 15, 2015 at 8:31 AM, Thomas Weise >>><[email protected]> >>> > > wrote: >>> > > >>> > > > It has not changed the operator execution model. State >>>serialization >>> is >>> > > > still synchronous, write to HDFS is taken out of the operator >>>thread. >>> > > > >>> > > > On Tue, Sep 15, 2015 at 8:18 AM, Amol Kekre >>><[email protected]> >>> > > wrote: >>> > > > >>> > > > > >>> > > > > Sent too soon. Has asynchronous checkpointing changed this? >>> > > > > >>> > > > > Amol >>> > > > > >>> > > > > Sent from my iPhone >>> > > > > >>> > > > > > On Sep 15, 2015, at 12:38 AM, Bhupesh Chawda < >>> > > [email protected]> >>> > > > > wrote: >>> > > > > > >>> > > > > > Hi All, >>> > > > > > >>> > > > > > Is it safe to assume that the checkpointed() and the >>> beginWindow() >>> > > > calls >>> > > > > > are sequenced? >>> > > > > > In other words, are these calls part of the same thread and >>>may >>> not >>> > > run >>> > > > > in >>> > > > > > parallel? >>> > > > > > >>> > > > > > Thanks. >>> > > > > > >>> > > > > > -- >>> > > > > > -Bhupesh >>> > > > > >>> > > > >>> > > >>> > >>>
