Since you not going to merge everything into one partition, you don't need to sort all messages across all partitions (because messages are sorted only within partition). I'd suggest splitting X partitions to Y groups and then merge source partitions within each group into their destination partition.
On Thu, Jan 16, 2020 at 10:20 AM Debraj Manna <subharaj.ma...@gmail.com> wrote: > Just to add when this operation will be going on no new data will be added > to original Kafka topic. I am trying to avoid buffering all data to a > temporary datastore to sort. > > On Thu, 16 Jan 2020, 23:14 Debraj Manna, <subharaj.ma...@gmail.com> wrote: > > > Hi > > > > I have a Kafka topic with X partitions. Each message has a timestamp, ts. > > Can someone suggest me some way of sorting all the messages (based on ts) > > across all partitions and putting it in a new topic with Y partitions (Y > < > > X ) using Kafka java client? > > > > Thanks > > > > >
