How can I auto refresh keyed producers to use new partitions as these partitions are added?
Try using the new producer under org.apache.kafka.clients.producer. Thanks, Neha On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry <mistry.p.bhav...@gmail.com> wrote: > I had different experience with expanding partition for new producer and > its impact. I only tried for non-key message. I would always advice to > keep batch size relatively low or plan for expansion with new java producer > in advance or since inception otherwise running producer code is impacted. > > Here is mail chain: > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3ccaoejijit4cgry97dgzfjkfvaqfduv-o1x1kafefbshgirkm...@mail.gmail.com%3E > > Thanks, > > Bhavesh > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shl...@viber.com> wrote: > > > Hmmm.. > > The Java producer example seems to ignore added partitions too... > > How can I auto refresh keyed producers to use new partitions as these > > partitions are added? > > > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shl...@viber.com> wrote: > > > > > One more thing: > > > I saw that the Python client is also unaffected by addition of > partitions > > > to a topic and that it continues to send requests only to the old > > > partitions. > > > is this also handled appropriately by the Java producer? Will he see > the > > > change and produce to the new partitions as well? > > > Shlomi > > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shl...@viber.com> > wrote: > > > > > >> No I don't see anything like that, the question was aimed at learning > if > > >> it is worthwhile to make the effort of reimplementing the Python > > producer > > >> in Java, I so I will not make all the effort just to be disappointed > > >> afterwards. > > >> understand I have nothing to worry about, so I will try to simulate > this > > >> situation in small scale... > > >> maybe 3 brokers, one topic with one partition and then add partitions. > > >> we'll see. > > >> thanks for clarifying. > > >> Oh, Good luck with Confluent!! > > >> :) > > >> > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede < > neha.narkh...@gmail.com > > > > > >> wrote: > > >> > > >>> The producer might get an error code if the leader of the partitions > > >>> being > > >>> reassigned also changes. However it should retry and succeed. Do you > > see > > >>> a > > >>> behavior that suggests otherwise? > > >>> > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <shl...@viber.com> > > wrote: > > >>> > > >>> > Hi All, > > >>> > I recently had an issue producing from python where expanding a > > cluster > > >>> > from 3 to 5 nodes and reassigning partitions forced me to restart > the > > >>> > producer b/c of KeyError thrown. > > >>> > Is this situation handled by the Java producer automatically or > need > > I > > >>> do > > >>> > something to have the java producer refresh itself to see the > > >>> reassigned > > >>> > partition layout and produce away ? > > >>> > Shlomi > > >>> > > > >>> > > >> > > >> > > > > > >
