You can also use stormkafkamon to track this stuff. Its not good for historical analysis like graphite/ganglia, but its good if you just want to see how things currently stand.
The original: https://github.com/otoolep/stormkafkamon This didn't work for us without some updates (incompatibility with the latest python-kafka dep). Here are those updates: https://github.com/BrightTag/stormkafkamon/commit/07eede9ec72329fe2cad893d087541b583e11148 (Our branch has a couple more things that parse the kafka messages with our format (which embeds a timestamp) to determine how long (in time) storm is behind... planning to clean that up soon so it can be a bit more reusable) https://github.com/BrightTag/stormkafkamon -Cody On Wed, May 28, 2014 at 4:50 AM, Danijel Schiavuzzi <[email protected]>wrote: > Yes, Trident Kafka spouts give you the same metrics. Take a look at the > code to find out what's available. > > > On Wed, May 28, 2014 at 3:55 AM, Tyson Norris <[email protected]> wrote: > >> Do Trident variants of kafka spouts do something similar? >> Thanks >> Tyson >> >> > On May 27, 2014, at 3:19 PM, "Harsha" <[email protected]> wrote: >> > >> > Raphael, >> > kafka spout sends metrics for kafkaOffset and kafkaPartition you >> can look at those by using LoggingMetrics or setting up a ganglia. Kafka >> uses its own zookeeper to store state info per topic & group.id you can >> look at kafka offsets using >> > kafka/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker >> > -Harsha >> > >> > >> >> On Tue, May 27, 2014, at 03:01 PM, Raphael Hsieh wrote: >> >> Is there a way to tell where in the kafka stream my topology is >> starting from? >> >> From my understanding Storm will use zookeeper in order to tell its >> place in the Kafka stream. Where can I find metrics on this ? >> >> How can I see how large the stream is? What how much data is sitting >> in the stream and what the most recent/oldest position is? >> >> >> >> Thanks >> >> >> >> -- >> >> Raphael Hsieh >> > > > > -- > Danijel Schiavuzzi > > E: [email protected] > W: www.schiavuzzi.com > T: +385989035562 > Skype: danijels7 > -- Cody A. Ray, LEED AP [email protected] 215.501.7891
