Hi, Pritam. I think the definition <http://But usually there is still a time difference in the propagation of different barriers> works for aligned checkpoint and unaligned checkpoint: "The alignment duration, which is defined as the time between receiving first and the last checkpoint barrier. " IIUC, Unaligned Checkpoints will help to speed up the flow of barriers greatly by putting barriers of input queue into output queue when they arrive at the operator. But usually there is still a time gap in the propagation of different barriers. For example, Barriers could be put into the output queue only when the current record is processed so that different operators have time gaps for barriers. So I agree with David that it's expected.
On Sat, Jun 3, 2023 at 2:30 AM David Anderson <dander...@apache.org> wrote: > I'm not 100% certain what "alignment duration" is measuring exactly in > the context of unaligned checkpoints -- however, even with unaligned > checkpointing each operator still has to wait until all of the > barriers are present in the operator's input queues. It doesn't have > to wait for the barriers to be aligned, but they do have to all be > present in order for the operator to know which in-flight messages to > include in the checkpoint. > > David > > On Fri, Jun 2, 2023 at 12:38 PM Pritam Agarwala > <pritamagarwala...@gmail.com> wrote: > > > > Hi All, > > > > > > I have enabled checkpointing in production. In peak hours, checkpointing > is falling due to timeout. When checked in sub-task level , I could see it > is taking some time for alignment and there is also a start delay. I think > the start delay will be due to back-pressure. > > As a first step , I have enabled unalignment checkpointing in my test > env. But I could still see it is taking time for alignment. Is this > expected behaviour. ?? > > > > Screenshot for your reference: > > In test environment: > > > > > > In prod environment : > > > > > > > > > > > > > > Thanks & Regards, > > Pritam > > > -- Best, Hangxiang.
