>> we've got a cron which iterates each topic+partition and writes an index of timestamps->byte offset
What is byte offset? Is this reliable to gauge the number of tuples being written to a topic when there are many different consumers writing to the same topic ? On Tue, Oct 7, 2014 at 4:35 PM, Neil Harkins <nhark...@gmail.com> wrote: > > On Tue, Oct 7, 2014 at 3:56 PM, Josh J <joshjd...@gmail.com> wrote: > >> Is there a way to monitor the rate of rates to a particular topic? I > wish > >> to monitor the frequency of incoming tuples in order to consume from the > >> topic in particular ways depending on the incoming write throughput. > > we've got a cron which iterates each topic+partition and > writes an index of timestamps->byte offset, so it was > easy to hook from there into graphite. but graphite couldn't > do 2000+ derivative()s for a reasonable pageload time, > so we put the pre-calculated write rate into graphite. > > what i'd very much like to have is per-topic read/write > stats by client ip. > > -neil >
