Evan,
That's correct. The Storm ZK consumer path for us is:
/{prefix}/{spout name}/10.x.x.x:9092:{partition}
and is a JSON blob. ConsumerOffsetChecker would then not work for this.
Mike
On Tue, Nov 20, 2012 at 12:11 PM, Evan Chan <e...@ooyala.com> wrote:
> Mike,
>
> I'm not sure the Storm-bundled kafka stores offsets in the same ZK
> locations as the regular Kafka consumer. Actually if you can verify the
> location that would be great, cuz I'm curious. Anyways the
> ConsumerOffsetChecker would not be able to help if the ZK locations were
> different.
>
> -Evan
>
> On Tue, Nov 20, 2012 at 8:03 AM, Mike Heffner <m...@librato.com> wrote:
>
> > I have not tried that yet, I was hoping to use an existing Ruby
> monitoring
> > process that we use to monitor several other existing resources. I also
> > don't want to make changes to the Kafka consumer code, as it's part of a
> > bundled package (Storm).
> >
> > Where does ConsumerOffsetChecker pull its information from? Shouldn't the
> > values from JMX match? Guess I might need to look at its source code to
> > figure out what it's doing.
> >
> >
> >
> >
> > On Tue, Nov 20, 2012 at 12:34 AM, Jun Rao <jun...@gmail.com> wrote:
> >
> > > Instead of using jmx, have you tried using ConsumerOffsetChecker to
> > figure
> > > out the consumer lag?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Nov 19, 2012 at 7:10 PM, Mike Heffner <m...@librato.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am trying to write a custom monitoring script for our Kafka setup
> and
> > > > would like some help understanding how to interpret the JMX
> attributes.
> > > >
> > > > In our setup, the consumers are writing their current offset to a
> path
> > in
> > > > ZK. This is the value they are getting back from a call
> > > > to SimpleConsumer.getOffsetsBefore(). A snapshot of this value looks
> > > like:
> > > >
> > > > {"offset"=>5338008447, "partition"=>2,
> "broker"=>{"host"=>"10.x.x.94",
> > > > "port"=>9092}, "topic"=>"mcommits"}
> > > >
> > > > Using the MX4J interface, I poll the
> > > > bean "kafka:type=kafka.logs.mcommits-2" on host 10.x.x.94 and get the
> > > > attribute values:
> > > >
> > > > {"CurrentOffset"=>506171524, "Name"=>"mcommits-2",
> > > > "NumAppendedMessages"=>10526508, "NumberOfSegments"=>4,
> > > "Size"=>2116784530}
> > > >
> > > > At the time both of these values were snapshotted, this consumer was
> > > close
> > > > to the end of the log file. In that case, I would expect both offsets
> > to
> > > be
> > > > fairly similar, however the consumer offset is >> the producer log
> > > offset,
> > > > which doesn't make sense.
> > > >
> > > > Clearly there is something I'm not understanding. How do I use the
> JMX
> > > > attributes to calculate how far behind the consumer is from the end
> of
> > > the
> > > > log file? In this scenario the consumer offset is >> both the
> > > CurrentOffset
> > > > value and the Size value. Is there a way of interpreting these values
> > > that
> > > > I'm not seeing?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Mike
> > > >
> > > > --
> > > >
> > > > Mike Heffner <m...@librato.com>
> > > > Librato, Inc.
> > > >
> > >
> >
> >
> >
> > --
> >
> > Mike Heffner <m...@librato.com>
> > Librato, Inc.
> >
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> e...@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>
--
Mike Heffner <m...@librato.com>
Librato, Inc.
