Other way could be to create multiple topics , as topics can be created on the fly. Producer can send data in a round robin fashion to individual topic as in load balanced way. Only problem i can see one might end up having lots of topics say 50-100 . Is this an issue??
-rks On Tue, May 15, 2012 at 10:51 PM, Edward Smith <esm...@stardotstar.org>wrote: > Speaking from inexperience: > > Wouldn't it be possible to just set the number of partitions on the > broker to a high number, and then use your own partitioner logic to > always select partition 0. Then, when you want more partitions, > modify the partitioner/producer to use more partitions? This way you > only have to restart producers and not the brokers. > > I don't actually understand entirely why the broker needs to know how > many partitions there will be. It seems he could just honor requests > to write to arbitrary partition numbers. > > Ed > > On Tue, May 15, 2012 at 12:22 PM, Jun Rao <jun...@gmail.com> wrote: > > Tarun, > > > > In 0.8, we plan to have an admin DDL that supports changing #partitions > of > > a topic and it won't require broker restarts. > > > > Jun > > > > On Tue, May 15, 2012 at 9:13 AM, Tarun Kumar <agrawal.taru...@gmail.com > >wrote: > > > >> Currently, I have one producer publishing messages on one topic (say > t1). I > >> have configured number of partitions on broker as 3 for topic t1 in > broker > >> config. At Consumer side, i have one consumer group with 3 consumers. I > >> have mapped these 3 consumers to 3 partitions (one to one mapping), each > >> consumer creates one map task and i store consumer's offsets in hdfs. > >> > >> But in future, if suppose producer starts producing messages at higher > >> speed, I would like to increase number of partitions for this topic (t1) > >> and number of consumers also for that consumer group. I want to avoid > >> broker restart. I was wondering if there is any way to do that. Broker > >> restart can be problematic in case i have multiple brokers. > >> > >> On Tue, May 15, 2012 at 9:35 PM, Jun Rao <jun...@gmail.com> wrote: > >> > >> > No, this is a config change on the broker and requires a broker > restart. > >> > > >> > Jun > >> > > >> > On Tue, May 15, 2012 at 8:27 AM, Tarun Kumar < > agrawal.taru...@gmail.com > >> > >wrote: > >> > > >> > > Is there any way to change the number of partitions topic wise on > the > >> fly > >> > > without restarting kafka server? > >> > > > >> > > >> >
