Please correct me if wrong, but I think it is really not hard constraint that one cannot have more consumers (from same group) than partitions on single topic - all the surplus consumers will not be assigned to consume any partition, but they can be there and as soon as one active consumer from same group goes offline (its connection to ZK is dropped), consumers from the group will be rebalanced so one passively waiting consumer will become active.
Kind regards, Stevo Slavic. On Wed, Apr 29, 2015 at 2:25 PM, David Corley <davidcor...@gmail.com> wrote: > If the 100 partitions are all for the same topic, you can have up to 100 > consumers working as part of a single consumer group for that topic. > You cannot have more consumers than there are partitions within a given > consumer group. > > On 29 April 2015 at 08:41, Nimi Wariboko Jr <n...@channelmeter.com> wrote: > > > Hi, > > > > I was wondering what options there are for horizontally scaling kafka > > consumers? Basically if I have 100 partitions and 10 consumers, and want > to > > temporarily scale up to 50 consumers, what options do I have? > > > > So far I've thought of just simply tracking consumer membership somehow > > (either through Raft or zookeeper's znodes) on the consumers. > > >
