The attributes (partition owner stats) that I see are as listed above:
(Broker/partition,fetch offset,consumer offset)
The JMX operations getOffsetLag and getLatestOffset return the lag and log
end respectively. The getConsumedOffset is equivalent to the consumer
offset in the above attribute.
Thanks,
Joel
On Wed, Nov 21, 2012 at 2:01 PM, Mike Heffner <m...@librato.com> wrote:
> Joel,
>
> We're using 0.7.1 right now. I see there's a 0.7.2 that was recently
> released, but we haven't updated to that yet.
>
> Do the mbean operations return different values from the attributes?
>
> Mike
>
>
> On Wed, Nov 21, 2012 at 1:47 PM, Joel Koshy <jjkosh...@gmail.com> wrote:
>
> > Hi Mike,
> >
> > What Kafka version are you using? I tried the latest trunk (0.7) and the
> > attributes I see are:
> > Broker/partition,fetch offset,consumer offset
> >
> > The values seem to be correct - and if you want to get the latest
> available
> > offset/consumer lag there are mbean operations (not attributes) that also
> > seem to work correctly.
> >
> >
> > On Wed, Nov 21, 2012 at 7:57 AM, Mike Heffner <m...@librato.com> wrote:
> >
> > > Well, I've currently "solved" the monitoring problem on our end by
> > skipping
> > > the JMX attributes and using the 'kafka-rb' Ruby gem to read the latest
> > > offset for the topic. This is the comparison of offsets read from the
> > > consumer, JMX, and the Ruby gem:
> > >
> > > Storm Consumer offset:
> > > ---------------------------------
> > > "offset"=>2847272176, "partition"=>2, "broker"=>{"host"=>"10.120.x.x",
> > > "port"=>9092}, "topic"=>"mcommits"}
> > >
> > > JMX Attributes:
> > > ---------------------
> > > attrs: {"CurrentOffset"=>162919578, "Name"=>"mcommits-2",
> > > "NumAppendedMessages"=>7285958, "NumberOfSegments"=>3,
> > "Size"=>1236661577}
> > >
> > > OFFSETS command sent from Ruby gem:
> > > ------------------------------------------------------------
> > > 10.120.x.x:mcommits:2: latest offset: 2847274728
> > >
> > > This is the output of the log file directory for this topic:partition:
> > >
> > > -rw-r--r-- 1 root root 536871010 2012-11-16 05:36
> > > 00000000001610613151.kafka
> > > -rw-r--r-- 1 root root 536870989 2012-11-20 14:28
> > > 00000000002147484161.kafka
> > > -rw-r--r-- 1 root root 162919578 2012-11-21 15:48
> > > 00000000002684355150.kafka
> > >
> > > So, the "latest offset" from the Ruby gem matches what I would expect
> to
> > > see -- only slightly ahead of the active consumer. AFAICT, the values
> > from
> > > JMX aren't usable to monitor this. Should I file a bug or feature
> request
> > > to publish an offset in the JMX attributes that matches the 'latest
> > offset'
> > > read from the Kakfa server?
> > >
> > > Cheers,
> > >
> > > Mike
> > >
> > >
> > > On Wed, Nov 21, 2012 at 12:08 AM, Jun Rao <jun...@gmail.com> wrote:
> > >
> > > > The attribute getCurrentOffset gives the log end offset. It's not
> > > > necessarily the log size though since older segments could be
> deleted.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Nov 20, 2012 at 1:12 PM, Mike Heffner <m...@librato.com>
> > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Do you have any idea on what the JMX attribute values on the beans
> "
> > > > > kafka:type=kafka.logs.{topic name}-{partition idx}" represent then?
> > It
> > > > > seems like these should correctly represent the current offsets of
> > the
> > > > > producer logs? They appeared to track correctly for a while, but
> once
> > > the
> > > > > log size grew, they seemed to no longer be correct. Is there
> > > potentially
> > > > a
> > > > > bug in these values are large log sizes?
> > > > >
> > > > > I can try the other interface, but it would be nice to know what's
> > > wrong
> > > > > with the current JMX values.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > > On Tue, Nov 20, 2012 at 12:12 PM, Jun Rao <jun...@gmail.com>
> wrote:
> > > > >
> > > > > > The tool gets the end offset of the log using getOffsetBefore and
> > the
> > > > > > consumer offset from ZK. It then calculates the lag.
> > > > > >
> > > > > > We do have a JMX for lag in ZookeeperConsumerConnector. The api
> is
> > > the
> > > > > > following, but you need to provide topic/brokerid/partitionid.
> > > > > >
> > > > > > /**
> > > > > > * JMX interface for monitoring consumer
> > > > > > */
> > > > > > trait ZookeeperConsumerConnectorMBean {
> > > > > > def getPartOwnerStats: String
> > > > > > def getConsumerGroup: String
> > > > > > def getOffsetLag(topic: String, brokerId: Int, partitionId:
> Int):
> > > > Long
> > > > > > def getConsumedOffset(topic: String, brokerId: Int,
> partitionId:
> > > > Int):
> > > > > > Long
> > > > > > def getLatestOffset(topic: String, brokerId: Int, partitionId:
> > > Int):
> > > > > Long
> > > > > > }
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Nov 20, 2012 at 8:03 AM, Mike Heffner <m...@librato.com>
> > > > wrote:
> > > > > >
> > > > > > > I have not tried that yet, I was hoping to use an existing Ruby
> > > > > > monitoring
> > > > > > > process that we use to monitor several other existing
> resources.
> > I
> > > > > also
> > > > > > > don't want to make changes to the Kafka consumer code, as it's
> > part
> > > > of
> > > > > a
> > > > > > > bundled package (Storm).
> > > > > > >
> > > > > > > Where does ConsumerOffsetChecker pull its information from?
> > > Shouldn't
> > > > > the
> > > > > > > values from JMX match? Guess I might need to look at its source
> > > code
> > > > to
> > > > > > > figure out what it's doing.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 20, 2012 at 12:34 AM, Jun Rao <jun...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Instead of using jmx, have you tried using
> > ConsumerOffsetChecker
> > > to
> > > > > > > figure
> > > > > > > > out the consumer lag?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <
> > m...@librato.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I am trying to write a custom monitoring script for our
> Kafka
> > > > setup
> > > > > > and
> > > > > > > > > would like some help understanding how to interpret the JMX
> > > > > > attributes.
> > > > > > > > >
> > > > > > > > > In our setup, the consumers are writing their current
> offset
> > > to a
> > > > > > path
> > > > > > > in
> > > > > > > > > ZK. This is the value they are getting back from a call
> > > > > > > > > to SimpleConsumer.getOffsetsBefore(). A snapshot of this
> > value
> > > > > looks
> > > > > > > > like:
> > > > > > > > >
> > > > > > > > > {"offset"=>5338008447, "partition"=>2,
> > > > > > "broker"=>{"host"=>"10.x.x.94",
> > > > > > > > > "port"=>9092}, "topic"=>"mcommits"}
> > > > > > > > >
> > > > > > > > > Using the MX4J interface, I poll the
> > > > > > > > > bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94
> and
> > > get
> > > > > the
> > > > > > > > > attribute values:
> > > > > > > > >
> > > > > > > > > {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> > > > > > > > > "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4,
> > > > > > > > "Size"=>2116784530}
> > > > > > > > >
> > > > > > > > > At the time both of these values were snapshotted, this
> > > consumer
> > > > > was
> > > > > > > > close
> > > > > > > > > to the end of the log file. In that case, I would expect
> both
> > > > > offsets
> > > > > > > to
> > > > > > > > be
> > > > > > > > > fairly similar, however the consumer offset is >> the
> > producer
> > > > log
> > > > > > > > offset,
> > > > > > > > > which doesn't make sense.
> > > > > > > > >
> > > > > > > > > Clearly there is something I'm not understanding. How do I
> > use
> > > > the
> > > > > > JMX
> > > > > > > > > attributes to calculate how far behind the consumer is from
> > the
> > > > end
> > > > > > of
> > > > > > > > the
> > > > > > > > > log file? In this scenario the consumer offset is >> both
> the
> > > > > > > > CurrentOffset
> > > > > > > > > value and the Size value. Is there a way of interpreting
> > these
> > > > > values
> > > > > > > > that
> > > > > > > > > I'm not seeing?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Mike Heffner <m...@librato.com>
> > > > > > > > > Librato, Inc.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Mike Heffner <m...@librato.com>
> > > > > > > Librato, Inc.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Mike Heffner <m...@librato.com>
> > > > > Librato, Inc.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Mike Heffner <m...@librato.com>
> > > Librato, Inc.
> > >
> >
>
>
>
> --
>
> Mike Heffner <m...@librato.com>
> Librato, Inc.
>
