In the new consumer design the partitioning algorithm can be customized by the user:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Would that fit your use case? Guozhang On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > May be right place to add such optimization within > https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ? > > -----Исходное сообщение----- > От: Guozhang Wang [mailto:wangg...@gmail.com] > Отправлено: Wednesday, May 28, 2014 8:57 PM > Кому: users@kafka.apache.org > Тема: Re: how to control assign policy for consumers > > With high-level consumers you cannot control the assignment of partitions > to consumers, and if partitions are located across all brokers each > consumer is likely to connect to each of the brokers. > > One (little hacky) thing you can do is to manually assign the partitions > to brokers such that each broker i of topic t is in broker i % 3, the for > consumer fetching partition i of topic t, it will only connect to broker i > % 3. By default the assignment is not aligned hence you will likely to have > TCP between each pair of consumer/broker. > > Guozhang > > > On Wed, May 28, 2014 at 8:15 AM, Рябков Алексей Николаевич < > a.ryab...@ntc-vulkan.ru> wrote: > > > How can I tell consumer to connect to one broker ...For example: > > - I have 3 topic and 3 broker...also each topic have 6 partitions... > > - I than start 6 consumers to listen this 3 topics...And in such > > case we can found from 6 (each consumer connect only to one broker) to > > 18 (each consumer connect to every broker) TCP connection .... How I > > can minimize network connections? > > > > With best regards, Aleksey Ryabkov > > > > > > > -- > -- Guozhang > -- -- Guozhang
