Re: programmatically get number of items in topic/partition

2014-10-05 Thread Joe Stein
Take a look into how Offset Requests are made and handled. You could do that to get the low and high watermarks for the stream. With the high and low watermark just subtract them to get total. This will be in the consumer api part of whatever language client your using. The specs for this are htt

Re: programmatically get number of items in topic/partition

2014-10-05 Thread Shlomi Hazan
actually this tool is not a 100% match to what I need, since it can only provide information on topics that have comsumers: Is there also another equivalent tool/method of querying topics that have no consumers ? in this case this tool will not help as it requires a group id as a mandatory paramete

Re: programmatically get number of items in topic/partition

2014-10-05 Thread Shlomi Hazan
Bingo. 10x!! On Wed, Oct 1, 2014 at 6:41 PM, chetan conikee wrote: > The other method is via command line > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > *groupName* > --zkconnect *zkServer:2181* > > Refer : > > https://cwiki.apache.org/confluence/display/KAFKA/System+Tool

Re: programmatically get number of items in topic/partition

2014-10-01 Thread chetan conikee
The other method is via command line bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group *groupName* --zkconnect *zkServer:2181* Refer : https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker https://apache.googlesource.com/kafka/+/0.8.0-bet

Re: programmatically get number of items in topic/partition

2014-10-01 Thread Gwen Shapira
Take a look at ConsumerOffsetChecker. It does just that: print the offset and lag for each consumer and partition. You can either use that class directly, or use it as a guideline for your implementation On Wed, Oct 1, 2014 at 2:10 AM, Shlomi Hazan wrote: > Hi, > How can I programmatically get t

programmatically get number of items in topic/partition

2014-10-01 Thread Shlomi Hazan
Hi, How can I programmatically get the number of items in a topic, pending for consumption? If no programmatic way is avail, what other method is available? Shlomi