Hi Vamsi, Kafka Streams uses the new 0.9.0 Java consumer as its underlying client, which do not have ZK dependency anymore. Instead it relies on a coordinator on the server side, one assigned for each consumer group to assign partitions among themselves.
You can find more details here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Guozhang On Fri, Feb 5, 2016 at 10:16 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Thanks Guozhang. > > Is the below true? > > 1. There is no central controller for the consuemrs which assigns the > partitions to the active consumers. > 2. Hence, the consumers each fight for the allocation by registering the > watch on active consumers (/ids path in zk). > > But how do the consumers finally arrive at concensus? Who ensures that the > consumer who took ownership actually is consuming data? We are noticing > scenarios where the rebalances are taking a lot of time and we have > (backoff-time * rebalance-retries) > zk-session-timeout. > > Would it be better to have a controller pattern followed by the consumers? > 1. Have a consumer controller which is leader elected > 2. The consumers then register their active state in zk > 3. The controller allocates the consumers the partitions depending on > strategies - ex: ceil(no-of-partitions/active-consumers) > 4. There is a background thread at the controller to monitor if the > allocated consumer is actually consuming the messages out of the partition > and if not, re-allocates them to others > > > > On Tue, Feb 2, 2016 at 10:46 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Vamsi, > > > > Consumer partition assignment is always within a single group. You can > find > > the java docs for the PartitionAssignor interface here: > > > > > > > https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/RangeAssignor.html > > > > StreamPartitionAssignor is one implementation of the interface which is > > used exclusively in Kafka Streams. > > > > Guozhang > > > > On Tue, Feb 2, 2016 at 1:37 AM, Achanta Vamsi Subhash < > > achanta.va...@flipkart.com> wrote: > > > > > Hi all, > > > > > > I am going through the partition assignment logic in the Kafka new > > > consumer. Is there any documentation for the logic implemented? It > would > > be > > > great if some one can write the algorithm for consumer partition > > assignment > > > (within a group and across groups). > > > > > > -- > > > Regards > > > Vamsi Subhash > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Regards > Vamsi Subhash > -- -- Guozhang
