Thank you that's helpful. So that means resource management of streams needs to a separate concern than that of Kafka connect
On Thu, Oct 25, 2018 at 3:10 PM Matthias J. Sax <matth...@confluent.io> wrote: > Streams is not designed to be run inside Connect, and this won't work. > > What you can do is, to import the data via connect into a "staging > topic" and then read this "staging topic" with a Kafka Streams > application and apply the transformations etc to write the data into the > actual target topics. > > > -Matthias > > On 10/25/18 2:34 PM, RK Sing wrote: > > Thank you Ryanne for the answer, My question is can I run Streams app > > inside Connect, what I mean is since Streams is a library, can I build a > > custom Source connector and use Streams in there . or using Streams in > > transforms?? > > We want to containerize Kafka connect as in wrap it in a docker, and > since > > all the Kafka-connect configs are stored in Kafka itself, so the > container > > can be stateless, we can spin up containers as and when needed, we want > > this approach to fetch data from microservices and golden gate oracle. > etc. > > > > > > > > > > On Thu, Oct 25, 2018 at 2:24 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > >> Dhurandar, definitely! Connect and Streams are both agnostic to how > their > >> workers are run. They aren't really platforms per se. You just need to > spin > >> up one or more workers and they do their thing. So a Streams app doesn't > >> run "inside" Connect, but you can certainly have Connect and Streams > >> workers talk to each other through Kafka topics. > >> > >> Ryanne > >> > >> On Thu, Oct 25, 2018 at 3:34 PM RK Sing <dhurandarg...@gmail.com> > wrote: > >> > >>> We have a requirement to do Single row transformations, basic joins, > >>> deduping and routing from the source to the destination Kafka topics. > >>> > >>> We want to use Kafka-connect as the platform which is running Kafka > >> stream > >>> inside. Has anyone used Kafkastreams inside Kafka connect ?? > >>> > >>> Is this pattern ok for Kafka-connect? > >>> > >>> regards > >>> dhurandar > >>> > >> > > > >
