Michael, It depends on how you define lag. 1) If you define lag as the total number of messages behind then burrow is a good tool as long as all your infrastructure is on 0.8.2, it basically works by inspecting the __consumer_offsets topic which was introduced in 0.8.2 (they said they were going to support <0.8.2 but i don't think that's a thing yet).
2. If you define lag in time units I would recommend sending a timestamp with the msg and doing some "manual" inspecting on the consumer end. -Jordan On Mon, Nov 16, 2015 at 9:16 AM, Michael Ravits <michaelr...@gmail.com> wrote: > Hi, > > I'd like to monitor consumer's lag. > Found this tool https://github.com/linkedin/Burrow. > But now realized that Samza is using it's own checkpointing mechanism. > > So question is what's the best way to monitor whether and how much the > consumer is lagging? > > On a related subject, I'd also like to monitor throughput per topic in > terms of messages per second and bytes per second. Should I query brokers > periodically, or maybe there is a better way? > > Thanks, > Michael > -- Jordan Shaw Full Stack Software Engineer PubNub Inc 1045 17th St San Francisco, CA 94107
