Hello! >>Which version of Kafka are you using? 1. I am using 0.8.1 >>How many partitions do you have in that topic? 2. In docs I found that number of partition must be equal or greater than number of consumers ... so I try 2 case: - 10k partition per broker - 20k partition per broker
And one more question - how to force rebalance partitions to consumers in borker (and how much it cost...) I found in doc that consumer can "Force itself to rebalance within in its consumer group" but can not find API call...So for now I just reconnect.... With best regards, Aleksey Ryabkov -----Исходное сообщение----- От: Jun Rao [mailto:jun...@gmail.com] Отправлено: Tuesday, May 27, 2014 7:29 AM Кому: users@kafka.apache.org Тема: Re: multi-broker topic rebalance Which version of Kafka are you using? How many partitions do you have in that topic? Thanks, Jun On Mon, May 26, 2014 at 7:14 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > > Hello! > > How can I rebalance topic within multiple brokers? > For example: > > 1. I have about 30k consumers... > > 2. I need to use 3 broker (10k consumer per broker) > > 3. I use multiple (around 10) producers with default balance strategy > Sometimes one broker have empty topic (consumer finished all message > on broker)...But another broker have a lot of messages in topic... > > How to resolve such situation: > -i cant just switch my consumers to another broker ( broker can not > handle more than 10k consumers....) -I can not find how to rebalance > topic within multiple brokers... > - I can not find how to setup limit for message per topic (or > partition) ... If such things exists I can handle such case when add > new message to broker in consumer side...and switch to another ... > > Thanks in advance, Aleksey Ryabkov >