Thanks for the heads up and information Todd! Cheers, -Kristoffer
On Wed, Jul 6, 2016 at 8:39 PM, Todd Palino <tpal...@gmail.com> wrote: > Yeah, it reflects the last offset in each partition all the time. The one > thing to be careful of is that it can be emitted from any replica > (whichever is the leader). So our tools that use the information always > look at all brokers and then take the larger non-null value. > > We've been using it for a while now, and it seems pretty safe. At least > none of our Kafka developers have complained about us doing it :) > > -Todd > > On Wednesday, July 6, 2016, Kristoffer Sjögren <sto...@gmail.com> wrote: > >> That's awesome! I can see the JMX bean [1] in our current 0.8.2 >> brokers and the number seems updated in real time per partition? I >> cannot find any official documentation on it? How safe is it to use? >> >> Also we're planning on upgrading to 0.10.0.0 - is it still available? >> >> Cheers, >> -Kristoffer >> >> [1] kafka.log:type=Log,name=LogEndOffset,topic=xxx,partition=xxx >> >> On Wed, Jul 6, 2016 at 5:36 PM, Todd Palino <tpal...@gmail.com >> <javascript:;>> wrote: >> > We do this through our monitoring agents by pulling it as a metric from >> the >> > LogEndOffset beans. By putting it into our metrics system we get a >> mapping >> > of timestamp to offset for every partition with (currently) 60 second >> > granularity. Useful for offset resets and other tasks. >> > >> > -Todd >> > >> > On Wednesday, July 6, 2016, Kristoffer Sjögren <sto...@gmail.com >> <javascript:;>> wrote: >> > >> >> Hi >> >> >> >> Is there a way to get the last offset written by all partitions of a >> >> topic programmatically using the 0.10.0.0 API? >> >> >> >> At the moment I use KafkaConsumer.seekToEnd as seen in this gist[1] >> >> but maybe there is a better, more efficient, way to do it? >> >> >> >> Cheers, >> >> -Kristoffer >> >> >> >> [1] https://gist.github.com/krisskross/a49e462bedb89505e372672cd81129ab >> >> >> > >> > >> > -- >> > *Todd Palino* >> > Staff Site Reliability Engineer >> > Data Infrastructure Streaming >> > >> > >> > >> > linkedin.com/in/toddpalino >> > > > -- > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino
