Hi Avani, not sure what you mean by creating your own code. Of course you can write an own Connector but especially in Source Connectors offset handling can challenging depending on the system. There are hundreds of different Connectors out there that can connect to existing systems. If you find one that suits your needs that's probably a better idea than starting from scratch. Typically there are lots of configuration options and for quite simple transformations you can use SMTs. To run Connectors, simply install some Connect instance, have the jars for your connectors installed and configure/deploy your Connector(s) using the Connect API.
Kind regards, Roman Am Di., 27. Juni 2023 um 11:26 Uhr schrieb Avani Panchal <avani.panc...@adit.com.invalid>: > Hi Roman, > > Thanks for replying, I want to ask one question: can I create my own code > to integrate data using Kafka connectors? > > Kind regards, > Avani Panchal > > On Tue, Jun 27, 2023 at 12:29 PM Roman Schmitz <roman.schm...@gmail.com> > wrote: > > > Hi Avani, > > > > totally agree - depending on the way you'd like to integrate your source > > that would be probably Debezium for CDC or JDBC source that can be used > > either in a "full-load" as well as in an incremental way. > > As a sink JDBC sink with the corresponding driver for the DB. Kindly > have a > > look at the Connect Documentation and some examples. > > If you'd like to test with runnable Docker-based examples, you can have a > > look at this Docker Playground Repo: > > https://github.com/vdesabou/kafka-docker-playground/tree/master/connect > > > > Kind regards, > > Roman > > > > Am Di., 27. Juni 2023 um 08:48 Uhr schrieb Avani Panchal > > <avani.panc...@adit.com.invalid>: > > > > > Hi Sagar, > > > > > > Thank you for your email. > > > > > > Thanks & Regards, > > > Avani Panchal > > > > > > > > > On Tue, Jun 27, 2023 at 12:15 PM Sagar <sagarmeansoc...@gmail.com> > > wrote: > > > > > > > Hi Avani, > > > > > > > > I already shared the documentation link for Kafka Connect. Let me > share > > > it > > > > again: https://kafka.apache.org/documentation/#connect > > > > > > > > Regarding the connector documentation, you should be able to find > them > > by > > > > just searching for JDBC source connector, JDBC sink connector and > > > Debezium > > > > connector for MySQL. > > > > > > > > Let me know if that works. > > > > > > > > Thanks! > > > > Sagar. > > > > > > > > > > > > > > > > On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal > > > > <avani.panc...@adit.com.invalid> wrote: > > > > > > > > > Hi Sagar, > > > > > > > > > > Thank you for the information, you solved our confusion. > > > > > I also saw lots of links for documentation on Kafka, but I am > > confused > > > > > which document I should use. > > > > > So can you share the proper link from where I can read the > documents. > > > > > > > > > > Thanks, > > > > > Avani Panchal > > > > > > > > > > > > > > > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sagarmeansoc...@gmail.com> > > > wrote: > > > > > > > > > > > Hey Avani, > > > > > > > > > > > > Kafka Connect <https://kafka.apache.org/documentation/#connect> > is > > > the > > > > > > tool > > > > > > to use when you want to stream data to/from Kafka via external > > > systems. > > > > > One > > > > > > would typically configure connectors which allow streaming data > > > to/from > > > > > > Kafka. There are 2 types of connectors: > > > > > > 1) Source Connectors: Which stream data from external systems > like > > > > > > databases etc to Kafka and > > > > > > 2) Sink Connectors: Which stream data from Kafka to external > > systems. > > > > > > > > > > > > Since you want to stream data from MySQL to SQL Server, with > Kafka > > > > > Connect > > > > > > it would be a 2 step process: > > > > > > > > > > > > 1) Capture changes from MySQL to Kafka using connectors like JDBC > > > > source > > > > > > connector or Debezium MySQL connector. > > > > > > 2) Once the data is in Kafka, you can use JDBC sink connectors to > > > > stream > > > > > > data from Kafka topics to the tables in SQL Server. > > > > > > > > > > > > Note that this is a very simplified view of how you can achieve > > your > > > > goal > > > > > > of streaming changes from MySQL to SQL Server and I would > recommend > > > > > reading > > > > > > the documentation of the individual connectors and the Kafka > > Connect > > > > > > framework to understand how to make it work for your usecase. > > > > > > > > > > > > Thanks for your interest on Apache Kafka! > > > > > > > > > > > > Thanks! > > > > > > Sagar. > > > > > > > > > > > > > > > > > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal > > > > > > <avani.panc...@adit.com.invalid> wrote: > > > > > > > > > > > > > Hi, > > > > > > > In my application I want to sync my client's data to my SQL > > > server. > > > > at > > > > > > > client place the database is MYSQL. > > > > > > > > > > > > > > How can I achieve this using Kafka? I read a lot of documents > > but I > > > > > don't > > > > > > > understand which setup I need and how I can achieve it. > > > > > > > > > > > > > > I was also wondering about "Book a demo with Kafka" but didn't > > find > > > > it. > > > > > > > > > > > > > > Please help me. > > > > > > > > > > > > > > Thank you, > > > > > > > Avani > > > > > > > > > > > > > > > > > > > > > > > > > > > >
