Re: Programmatic Kafka version detection/extraction?

with this value, or maybe an MBean or some other way to get to this info? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ -- Joey Echeverria

Re: Consumer with more stable partition assignments?

I don't have a solution, but I thought I'd chime in with interest in finding a solution to this problem. We have a use case where we're partitioning the dataset we write to according to Kafka partitions and having to close all writers and re-open after a rebalance is a pain point. -Joey > On

Partition ownership with high-level consumer

Hi! Is there a way to track current partition ownership when using the high-level consumer? It looks like the rebalance callback only tells me the partitions I'm (potentially) losing. -Joey

Re: Partition ownership with high-level consumer

But nothing in the API? -Joey On Tue, Oct 6, 2015 at 3:43 PM, Gwen Shapira <g...@confluent.io> wrote: > Zookeeper will have this information under /consumers//owners > > > > On Tue, Oct 6, 2015 at 12:22 PM, Joey Echeverria <j...@rocana.com> wrote: > > >

Re: Partition ownership with high-level consumer

: > I don't think so. AFAIK, even the new API won't send this information to > every consumer, because in some cases it can be huge. > > > > On Tue, Oct 6, 2015 at 1:44 PM, Joey Echeverria <j...@rocana.com> wrote: > > > But nothing in the API? > > > >

Handling long commits during a rebalance

We've been playing around with the new Consumer API and have it an unfortunate bump in the road. When our onPartitionsRevoked() callback is called we'd like to be able to commit any data that we were processing to stable storage so we can then commit the offsets back to Kafka. This way we don't

Re: Handling long commits during a rebalance

The consumer thread would check to > see if there were offsets that needed to committed, if so it would commit > them. > > It was a bunch of work to get it working and it's still not perfect, but > it's getting the job done. > > -craig > > On Tue, Jul 26, 2016 at 3:13 PM, Joe

Re: Handling long commits during a rebalance

voked occurs the blocking queue is cleared...again may not > ideal, but works for this use case. > > On Tue, Jul 26, 2016 at 8:53 PM, Joey Echeverria <j...@rocana.com> wrote: > >> That's the direction we're looking at for normal commit processing, >> but how do you