"If your system is stateless and the transformations are not interdependent" then I would just look at using Kafka Connect's Single Message Transform (SMT) feature.
-hans /** * Hans Jespersen, Director Systems Engineering, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Tue, Mar 13, 2018 at 9:18 AM, Jacob Sheck <shec0...@gmail.com> wrote: > If you are augmenting streaming data with dimensional data, or if you can > transform your data with a map, filter or join operation Streams will be a > good option. If your system is stateless and the transformations are not > interdependent, you may want to look into using one of the queue > technologies like AcitveMQ. > > On Tue, Mar 13, 2018 at 10:00 AM Sameer Rahmani <lxsame...@gmail.com> > wrote: > > > Thanks Jacob. I'm using kafka as a distributed queue and my data pipeline > > is a several components which connected together via a stream > abstraction. > > each component has a input and output stream. Basically the source of > > this pipeline can be anything and the output can be anything to. > > > > On Tue, Mar 13, 2018 at 1:27 PM, Jacob Sheck <shec0...@gmail.com> wrote: > > > > > Sameer when you say that you need to "consume from and produce to a > > topic" > > > to me that seems like a good fit for Kafka Streams. Streaming your > data > > > out of Kafka for a transform and back in has some fundamental costs and > > > operational challenges involved. Are the events in your stream > > stateless? > > > If it isn't stateless streams will ensure a consistent playback of > events > > > if needed. Without knowing more about your pipeline it is hard to make > > > recommendations. Are you possibly using Kafka as a distributed queue? > > > > > > On Tue, Mar 13, 2018 at 6:29 AM Sameer Rahmani <lxsame...@gmail.com> > > > wrote: > > > > > > > Hi folks, > > > > I need to consume from and produce to a topic. I have my own data > > > pipeline > > > > to process the data. > > > > So I was wondering beside the stores and StreamDSL what does Kafka > > > Streams > > > > brings to the table > > > > that might be useful to me ? > > > > > > > > > >
