Hello Max, Thanks for the answer. My guess was that they are acknowledged at completion of Flink's checkpoints, but wanted to make sure since that doesn't explain my problem.
Whenever a subscription is nearly empty the job gets slower overall and the Flink's checkpoints start taking much more time (thrice or more) even though their state is much smaller, and of course, there always seem to be messages cycling over and over again. If you have any clue at all why this might be, let me know. Thanks for the help, Encho On Tue, Sep 11, 2018 at 1:45 PM Maximilian Michels <[email protected]> wrote: > Hey Encho, > > The Flink Runner acknowledges messages through PubSubIO's > `CheckpointMark#finalizeCheckpoint()` method. > > The Flink Runner wraps the PubSubIO source via the > UnboundedSourceWrapper. When Flink takes a checkpoint of the running > Beam streaming job, the wrapper will retrieve the CheckpointMarks from > the PubSubIO source. > > When the Checkpoint is completed, there is a callback which informs the > wrapper (`notifyCheckpointComplete()`) and calls `finalizeCheckpoint()` > on all the generated CheckpointMarks. > > Hope that helps debugging your problem. I don't have an explanation why > this doesn't work for the last records in your PubSub queue. It > shouldn't make a difference for how the Flink Runner does checkpointing. > > Best, > Max > > On 10.09.18 18:17, Encho Mishinev wrote: > > Hello, > > > > I am using Flink runner with Apache Beam 2.6.0. I was wondering if there > > is information on when exactly the runner acknowledges a pubsub message > > when reading from PubsubIO? > > > > My problem is that whenever there are a few messages left in a > > subscription my streaming job never really seems to acknowledge them > > all. For example is a subscription has 100,000,000 messages in total, > > the job will go through about 99,990,000 and then keep reading the last > > few thousand and seemingly never acknowledge them. > > > > Some clarity on when the acknowledgement happens in the pipeline might > > help me debug this problem. > > > > Thanks! >
