Hi,
I am trying to write a custom monitoring script for our Kafka setup and
would like some help understanding how to interpret the JMX attributes.
In our setup, the consumers are writing their current offset to a path in
ZK. This is the value they are getting back from a call
to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks like:
{"offset"=>5338008447, "partition"=>2, "broker"=>{"host"=>"10.x.x.94",
"port"=>9092}, "topic"=>"mcommits"}
Using the MX4J interface, I poll the
bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
attribute values:
{"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
"NumAppendedMessages"=>10526508, "NumberOfSegments"=>4, "Size"=>2116784530}
At the time both of these values were snapshotted, this consumer was close
to the end of the log file. In that case, I would expect both offsets to be
fairly similar, however the consumer offset is >> the producer log offset,
which doesn't make sense.
Clearly there is something I'm not understanding. How do I use the JMX
attributes to calculate how far behind the consumer is from the end of the
log file? In this scenario the consumer offset is >> both the CurrentOffset
value and the Size value. Is there a way of interpreting these values that
I'm not seeing?
Thanks,
Mike
--
Mike Heffner <m...@librato.com>
Librato, Inc.
