Hi All, Thanks a lot for the information and suggestion!
Eleanore On Tue, May 12, 2020 at 3:24 PM Malcolm McFarland <mmcfarl...@cavulus.com> wrote: > These ideas are specific to Samza and ymmv in how they apply to other > processing frameworks, but we use a couple of custom tools to keep tabs on > processing lag: > > - one is a produce/consume timestamp comparison tool which utilizes writes > a message production timestamps out to ZooKeeper on a per-partition basis; > then in our stream processor, Samza, we then write out a consumption > timestamp for the same partition to ZooKeeper, and we can use these > differences (with some compensations for partition offset differences) to > see what our processing lag is; > - we also have a custom offset/checkpoint comparison tool which ingests > Samza's checkpoint topic and compares it against the latest offsets on a > per-partition basis to know how far behind each partition is in processing > messages (this also doubles as a checkpoint-properties file generator which > we can use to rebuild the checkpoint topic if it gets too large) > > These two tools have been invaluable in helping us monitor our Samza > processing clusters. > > Cheers, > Malcolm McFarland > Cavulus > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any > unauthorized or improper disclosure, copying, distribution, or use of the > contents of this message is prohibited. The information contained in this > message is intended only for the personal and confidential use of the > recipient(s) named above. If you have received this message in error, > please notify the sender immediately and delete the original message. > > > On Mon, May 11, 2020 at 5:23 PM Eleanore Jin <eleanore....@gmail.com> > wrote: > > > Hi community, > > > > I just wonder what is the difference between the consumer lag reported by > > Kafka client and the consumer lag reported by burrow? > > > > Thanks a lot! > > Eleanore > > >
