Thanks Chad! if we want to consume from multiple topics and persist to a database , can i go with a consumer and lookup the record and update .Requirement is to consume from item topic and price topic and create a record in postgress . Both topic have item id in message which is the key in postgress database . Can this be done with a simple consumer ?
On Thu, Jan 13, 2022 at 11:11 AM Chad Preisler <chad.preis...@gmail.com> wrote: > Yes Kafka streams can be used to do this. There are probably several ways > to implement this. We did something like this in Java using a groupByKey() > and reduce() functions. The three topics we wanted to combine into one > topic had different schemas and different java class types. So to combine > them together into one aggregated object we did the following. > > - Create a class with data members of the three objects we wanted to > combine. Let's call it AggregateObj. > - Create a KStream for each topic we wanted to combine. > - For each KStream use a map function that creates and outputs an > AggregateObj setting the input stream object to the correct data member on > the AggregateObj. > - Create an intermediate topic to write individual AggregateObj from each > KStream. > - Create a stream to read the intermediate topic and use the groupByKey() > and reduce() function to create one AggregateObj that has all the parts. > Output that result to the final combined output stream using > toStream().to(). > > We did all of this in one application. You may be able to accomplish the > same thing using aggregate a different way or you may be able to use left > join methods to accomplish the same thing. I can't share the code. Sorry. > > On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sreekumar.prad...@gmail.com> > wrote: > > > Hi , > > I have a requirement to stream item details to specific destinations . > > There are three different kafka streams , one for item info, second for > > item price and promotions and third for item availability . > > I want to join all these info and produce a single message containing > > item,price and availability . > > Can kafka streams be leveraged for this , if all the messages across > three > > topics can be joined using a coming item identifier . Also its not > > necessary that all three topics have data . > > For example, if item setup is done and no inventory is allocated, only > item > > and price topics will have data . Also any good pointers to see a sample > > app for joining multiple streams and producing a single message to a new > > topic. > > Thanks > > Pradeep > > >
