Hi RK, You may find this blog post helpful for incorporating Streams with Connect:
https://yokota.blog/2018/04/23/embedding-kafka-connect-in-kafka-streams-ksql/ I've also cc'ed Robert, the author of this blog post if you are interested in his code and want to learn more. Guozhang On Tue, Oct 30, 2018 at 8:54 PM RK Sing <dhurandarg...@gmail.com> wrote: > Thank you that's helpful. > > So that means resource management of streams needs to a separate concern > than that of Kafka connect > > On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax <matth...@confluent.io> > wrote: > > > Streams is not designed to be run inside Connect, and this won't work. > > > > What you can do is, to import the data via connect into a "staging > > topic" and then read this "staging topic" with a Kafka Streams > > application and apply the transformations etc to write the data into the > > actual target topics. > > > > > > -Matthias > > > > On 10/25/18 2:34 PM, RK Sing wrote: > > > Thank you Ryanne for the answer, My question is can I run Streams app > > > inside Connect, what I mean is since Streams is a library, can I build > a > > > custom Source connector and use Streams in there . or using Streams in > > > transforms?? > > > We want to containerize Kafka connect as in wrap it in a docker, and > > since > > > all the Kafka-connect configs are stored in Kafka itself, so the > > container > > > can be stateless, we can spin up containers as and when needed, we want > > > this approach to fetch data from microservices and golden gate oracle. > > etc. > > > > > > > > > > > > > > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > >> Dhurandar, definitely! Connect and Streams are both agnostic to how > > their > > >> workers are run. They aren't really platforms per se. You just need to > > spin > > >> up one or more workers and they do their thing. So a Streams app > doesn't > > >> run "inside" Connect, but you can certainly have Connect and Streams > > >> workers talk to each other through Kafka topics. > > >> > > >> Ryanne > > >> > > >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing <dhurandarg...@gmail.com> > > wrote: > > >> > > >>> We have a requirement to do Single row transformations, basic joins, > > >>> deduping and routing from the source to the destination Kafka topics. > > >>> > > >>> We want to use Kafka-connect as the platform which is running Kafka > > >> stream > > >>> inside. Has anyone used Kafkastreams inside Kafka connect ?? > > >>> > > >>> Is this pattern ok for Kafka-connect? > > >>> > > >>> regards > > >>> dhurandar > > >>> > > >> > > > > > > > > -- -- Guozhang
