Re: How to balance messages in kafka topics with newly added partitions?

2019-01-27 Thread Hans Jespersen
Yes but I find this even easier to do with KSQL. CREATE STREAM OUTPUTTOPIC AS SELECT * FROM INPUTTOPIC; There are similar examples like this that also filter messages while copying, or change the message format while copying on the KSQL Recipe page here

Re: How to balance messages in kafka topics with newly added partitions?

2019-01-27 Thread Ryanne Dolan
You can use MirrorMaker to copy data between topics. Ryanne On Sun, Jan 27, 2019, 7:12 AM jaaz jozz Thanks, Sönke > Is there any available kafka tool to move messages between topics? > > On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau > wrote: > > > Hi Jazz, > > > > I'm afraid the only way of

Re: How to balance messages in kafka topics with newly added partitions?

2019-01-27 Thread jaaz jozz
Thanks, Sönke Is there any available kafka tool to move messages between topics? On Sun, Jan 27, 2019 at 2:40 PM Sönke Liebau wrote: > Hi Jazz, > > I'm afraid the only way of rebalancing old messages is indeed to > rewrite them to the topic - thus creating duplication. > Once a message has been

Re: How to balance messages in kafka topics with newly added partitions?

2019-01-27 Thread Sönke Liebau
Hi Jazz, I'm afraid the only way of rebalancing old messages is indeed to rewrite them to the topic - thus creating duplication. Once a message has been written to a partition by Kafka this assignment is final, there is no way of moving it to another partition. Changing the partition count of

How to balance messages in kafka topics with newly added partitions?

2019-01-27 Thread jaaz jozz
Hello, I have kafka cluster with certain topic that had too few partitions, so a large backlog of messages was collected. After i added additional partitions, only the newly messages balanced between all the new partitions. What is the preferred way to balance the "old" backlog of messages