Hi All, I have some confusion regarding the attributes which control the checkpointing in Apex.
As per my understanding, there are three attributes which play a role in deciding when to perform checkpointing: 1. OperatorContext.APPLICATION_WINDOW_COUNT (How many streaming windows form an app window) 2. OperatorContext.CHECKPOINT_WINDOW_COUNT (Indicates the optimal checkpoint boundary) 3. DagContext.CHECKPOINT_WINDOW_COUNT (how many streaming windows form a checkpoint window) I have the following doubts: - Let's consider the case where checkpointing always happens at an Application window boundary. This implies OperatorContext.APPLICATION_WINDOW_COUNT = OperatorContext.CHECKPOINT_WINDOW_COUNT. The engine will always insert a "checkpoint marker" tuple after the DagContext.CHECKPOINT_WINDOW_COUNT streaming windows are done. However, if this marker arrives at a time when the application window is still not over, then the checkpoint will be delayed till the end of the application window. Is this understanding correct? - One inconsistency that I have observed: When OperatorContext.APPLICATION_WINDOW_COUNT = OperatorContext.CHECKPOINT_WINDOW_COUNT = 1, the first checkpoint happens after the 58th window instead of 59th window (0 indexed). The following checkpoints however happen at the intended 118th, 178th etc. windows. This is for the default case of 60 windows checkpoint. In other cases, when OperatorContext.APPLICATION_WINDOW_COUNT = OperatorContext.CHECKPOINT_WINDOW_COUNT > 1 (and a perfect divisor of DagContext.CHECKPOINT_WINDOW_COUNT), it happens at the 59th window, 119th window and so on. Is this behaviour expected? - Moving on to the other case of allowing checkpointing within application windows; here OperatorContext.APPLICATION_WINDOW_COUNT may not be equal to OperatorContext.CHECKPOINT_WINDOW_COUNT. Similar to the previous case, the "checkpoint marker" tuple arrives at the end of DagContext.CHECKPOINT_WINDOW_TUPLES. In this case, there is no restriction on when not to checkpoint. So what is the role of OperatorContext.CHECKPOINT_WINDOW_COUNT in this case? Thanks. -Bhupesh
