Sorry, I should say only partition 1 had something at first, then zero: Toms-iMac:betwave-server tomdearman$ /Users/tomdearman/software/kafka_2.11-0.10.0.0/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --describe --group voidbridge-oneworks-dummy GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER voidbridge-oneworks-dummy integration-oneworks-dummy 2 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 7 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 12 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 17 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 4 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 9 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 14 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 19 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 1 3 3 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 6 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 11 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 16 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 3 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 8 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 13 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 18 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 0 10 0 -10 integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 5 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 10 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 15 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 Toms-iMac:betwave-server tomdearman$ /Users/tomdearman/software/kafka_2.11-0.10.0.0/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --describe --group voidbridge-oneworks-dummy GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER voidbridge-oneworks-dummy integration-oneworks-dummy 2 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 7 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 12 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 17 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-3_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 4 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 9 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 14 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 19 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-5_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 1 3 3 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 6 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 11 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 16 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-2_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 3 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 8 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 13 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 18 unknown 0 unknown integration-oneworks-dummy-voidbridge-oneworks-dummy-4_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 0 1 1 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 5 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 10 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113 voidbridge-oneworks-dummy integration-oneworks-dummy 15 0 0 0 integration-oneworks-dummy-voidbridge-oneworks-dummy-1_/10.100.0.113
> On 8 Jul 2016, at 17:20, Tom Dearman <tom.dear...@gmail.com> wrote: > > When you say ‘for the first partition’ do you literally mean partition zero, > or you mean any partition. It is true that when I had only 1 user there were > only messages on partition 15 but the second user happened to go to partition > zero. Is it the case that partition zero must have a consumer commit? > >> On 8 Jul 2016, at 17:16, Todd Palino <tpal...@gmail.com> wrote: >> >> If you open up an issue on the project, I'd be happy to dig into this in >> more detail if needed. Excluding the ZK offset checking, Burrow doesn't >> enumerate consumer groups - it learns about them from offset commits. It >> sounds like maybe your consumer had not committed offsets for the first >> partition (at least not after Burrow was started). >> >> -Todd >> >> On Friday, July 8, 2016, Tom Dearman <tom.dear...@gmail.com> wrote: >> >>> Todd, >>> >>> Thanks for that I am taking a look. >>> >>> Is there a bug whereby if you only have a couple of messages on a topic, >>> both with the same key, that burrow doesn’t return correct info. I was >>> finding that http://localhost:8100/v2/kafka/betwave/consumer < >>> http://localhost:8100/v2/kafka/betwave/consumer> was returning a message >>> with empty consumers until I put on another message with a different key, >>> i.e. a minimum of 2 partitions with something in them. I know this is not >>> very like production, but on my local this I was only testing with one user >>> so get just one partition filled. >>> >>> Tom >>>> On 6 Jul 2016, at 18:08, Todd Palino <tpal...@gmail.com <javascript:;>> >>> wrote: >>>> >>>> Yeah, I've written dissertations at this point on why MaxLag is flawed. >>> We >>>> also used to use the offset checker tool, and later something similar >>> that >>>> was a little easier to slot into our monitoring systems. Problems with >>> all >>>> of these is why I wrote Burrow (https://github.com/linkedin/Burrow) >>>> >>>> For more details, you can also check out my blog post on the release: >>>> >>> https://engineering.linkedin.com/apache-kafka/burrow-kafka-consumer-monitoring-reinvented >>>> >>>> -Todd >>>> >>>> On Wednesday, July 6, 2016, Tom Dearman <tom.dear...@gmail.com >>> <javascript:;>> wrote: >>>> >>>>> I recently had a problem on my production which I believe was a >>>>> manifestation of the issue kafka-2978 (Topic partition is not sometimes >>>>> consumed after rebalancing of consumer group), this is fixed in 0.9.0.1 >>> and >>>>> we will upgrade our client soon. However, it made me realise that I >>> didn’t >>>>> have any monitoring set up on this. The only thing I can find as a >>> metric >>>>> is the >>>>> >>> kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+), >>>>> which, if I understand correctly, is the max lag of any partition that >>> that >>>>> particular consumer is consuming. >>>>> 1. If I had been monitoring this, and if my consumer was suffering from >>>>> the issue in kafka-2978, would I actually have been alerted, i.e. since >>> the >>>>> consumer would think it is consuming correctly would it not have updated >>>>> the metric. >>>>> 2. There is another way to see offset lag using the command >>>>> /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server >>>>> 10.10.1.61:9092 --describe —group consumer_group_name and parsing the >>>>> response. Is it safe or advisable to do this? I like the fact that it >>>>> tells me each partition lag, although it is also not available if no >>>>> consumer from the group is currently consuming. >>>>> 3. Is there a better way of doing this? >>>> >>>> >>>> >>>> -- >>>> *Todd Palino* >>>> Staff Site Reliability Engineer >>>> Data Infrastructure Streaming >>>> >>>> >>>> >>>> linkedin.com/in/toddpalino >>> >>> >> >> -- >> *Todd Palino* >> Staff Site Reliability Engineer >> Data Infrastructure Streaming >> >> >> >> linkedin.com/in/toddpalino >