Instead of using jmx, have you tried using ConsumerOffsetChecker to figure
out the consumer lag?
Thanks,
Jun
On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <m...@librato.com> wrote:
> Hi,
>
> I am trying to write a custom monitoring script for our Kafka setup and
> would like some help understanding how to interpret the JMX attributes.
>
> In our setup, the consumers are writing their current offset to a path in
> ZK. This is the value they are getting back from a call
> to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
>
> {"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
> "port"=>9092}, "topic"=>"mcommits"}
>
> Using the MX4J interface, I poll the
> bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> attribute values:
>
> {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
>
> At the time both of these values were snapshotted, this consumer was close
> to the end of the log file. In that case, I would expect both offsets to be
> fairly similar, however the consumer offset is >> the producer log offset,
> which doesn't make sense.
>
> Clearly there is something I'm not understanding. How do I use the JMX
> attributes to calculate how far behind the consumer is from the end of the
> log file? In this scenario the consumer offset is >> both the CurrentOffset
> value and the Size value. Is there a way of interpreting these values that
> I'm not seeing?
>
>
> Thanks,
>
> Mike
>
> --
>
> Mike Heffner <m...@librato.com>
> Librato, Inc.
>
