Thanks Daniyar for replying. Do kafka streams have any apis to do the partitioning and grouping that you are suggesting?
Also if I have to merge everything into a single partition what should be the efficient way to do this? On Fri, Jan 17, 2020 at 6:03 AM Daniyar Kulakhmetov <dkulakhme...@liftoff.io> wrote: > Since you not going to merge everything into one partition, you don't need > to sort all messages across all partitions (because messages are sorted > only within partition). > I'd suggest splitting X partitions to Y groups and then merge source > partitions within each group into their destination partition. > > > On Thu, Jan 16, 2020 at 10:20 AM Debraj Manna <subharaj.ma...@gmail.com> > wrote: > > > Just to add when this operation will be going on no new data will be > added > > to original Kafka topic. I am trying to avoid buffering all data to a > > temporary datastore to sort. > > > > On Thu, 16 Jan 2020, 23:14 Debraj Manna, <subharaj.ma...@gmail.com> > wrote: > > > > > Hi > > > > > > I have a Kafka topic with X partitions. Each message has a timestamp, > ts. > > > Can someone suggest me some way of sorting all the messages (based on > ts) > > > across all partitions and putting it in a new topic with Y partitions > (Y > > < > > > X ) using Kafka java client? > > > > > > Thanks > > > > > > > > >
