Speaking from inexperience: Wouldn't it be possible to just set the number of partitions on the broker to a high number, and then use your own partitioner logic to always select partition 0. Then, when you want more partitions, modify the partitioner/producer to use more partitions? This way you only have to restart producers and not the brokers.
I don't actually understand entirely why the broker needs to know how many partitions there will be. It seems he could just honor requests to write to arbitrary partition numbers. Ed On Tue, May 15, 2012 at 12:22 PM, Jun Rao <jun...@gmail.com> wrote: > Tarun, > > In 0.8, we plan to have an admin DDL that supports changing #partitions of > a topic and it won't require broker restarts. > > Jun > > On Tue, May 15, 2012 at 9:13 AM, Tarun Kumar <agrawal.taru...@gmail.com>wrote: > >> Currently, I have one producer publishing messages on one topic (say t1). I >> have configured number of partitions on broker as 3 for topic t1 in broker >> config. At Consumer side, i have one consumer group with 3 consumers. I >> have mapped these 3 consumers to 3 partitions (one to one mapping), each >> consumer creates one map task and i store consumer's offsets in hdfs. >> >> But in future, if suppose producer starts producing messages at higher >> speed, I would like to increase number of partitions for this topic (t1) >> and number of consumers also for that consumer group. I want to avoid >> broker restart. I was wondering if there is any way to do that. Broker >> restart can be problematic in case i have multiple brokers. >> >> On Tue, May 15, 2012 at 9:35 PM, Jun Rao <jun...@gmail.com> wrote: >> >> > No, this is a config change on the broker and requires a broker restart. >> > >> > Jun >> > >> > On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar <agrawal.taru...@gmail.com >> > >wrote: >> > >> > > Is there any way to change the number of partitions topic wise on the >> fly >> > > without restarting kafka server? >> > > >> > >>
