Re: some producers stuck when one broker is bad

frankly I don't know exactly what went BAD for that broker. process is still UP. On Wed, Sep 9, 2015 at 10:10 AM, Mayuresh Gharat wrote: > 1) any suggestion on how to identify the bad broker(s)? > ---> At Linkedin we have alerts that are setup using our internal scripts > for detecting if a brok

Can current co-ordination protocol cause data loss or delays in consumption?

Hi, we plan to use kafka as main message bus and we strongly rely on two features: - at-least-once delivery guarantee - using groups for load balancing between several consumers But there are some problems described in https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design ,

mirror maker mixed with other producers

Hi all, I think I know the answer but just wanted to verify with others.. If I have producers writing into topic X in one cluster A, I can run Mirror Maker to write into the same topic X from another cluster B. They are all producers after all, right? Any complications? Thanks, Foo

Re: automatically consume from all topics

You can create message streams using regex that includes all topics. The beauty of regex is that any new topic created will be automatically consumed as long as the name of the topic matches the regex You check the method createMessageStreamsByFilter in high level API On Thu, Sep 10, 2015 at 11:0

Re: Partition Consumer(s)

So, the general scalability approach with kafka is to add more partitions to scale. If you are using consumer groups and the High Level Consumer API, redistribution of partitions is automatic on a failover of a member of a consumer group. But, the High level consumer doesn¹t allow a configuratio

Re: automatically consume from all topics

Hi, I'm not sure about the high level consumer but I maintain a kafka consumer that can add and remove topics dynamically. https://github.com/gerritjvv/kafka-fast see https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java if you're using java/scala On T

Partition Consumer(s)

Hi Everybody, I have 2 question regarding the way consumers, consume messages of a partition. - * Is it possible to configure Kafka to allow concurrent message consumption from one partition concurrently? The order is not my concern at all.* I couldn't find any way to that b

Re: automatically consume from all topics

So, there are several ways to do this. Lets assume the goal is to add more topics to the application at runtime. And that this app is currently written to be distributed via the magic of consumer groups. Sadly, I don¹t think the High level consumer is well designed for this particular use case.

automatically consume from all topics

Hello, Is there a simple way to set up a consumer that automatically picks up all the topics for all the partitions, dynamically extending its range as new topics get created? The underlying idea is that we want to have a few over-arching consumers (I'm aware that's not great for the scalabili