Hello,

We are currently building our data-pipeline using Confluent and as part of this 
implementation, we have written couple of Kafka Connect Sink Connectors for 
Azure and MS SQL server. To provide some more context, we are planning to use 
SQL Server's Change Data Capture feature to track changes happening on 
individual tables in the source database and then use JDBC connector's "QUERY" 
mode to fetch the changed data and push it to Kafka topics. We had to go down 
this route because we cannot guarantee the presence of Identity or Timestamp 
columns in our source tables so we are relying on the timestamp columns present 
in the system tables created by Change Data Capture (trans_begin_time) and use 
it along with "QUERY" mode in JDBC connector. However, this design requires us 
to run one JDBC connector instance per table.

On the consumer side, once the data is on Kafka, we have written an Azure 
connector to push the data from Kafka topics onto Azure Blob Storage and 
another one to push data to MS SQL server.

Now coming to the actual question :) , the above design leads to a number of 
Kafka Connect workers, connectors and tasks running on our distributed cluster. 
We are planning to run multiple workers per host and have them share the tasks 
from our 3 connectors (JDBC, Azure and SQL Server). How can we monitor these 
connectors and tasks in a production environment? How can we make sure that 
workers are restarted once they are killed? How can we identify the failing 
tasks on different workers?Are there any best practices on monitoring Kafka 
Connect and managing workers/connectors/tasks?

We are close to finishing our development and deploying this to test and 
production environments and it is very important that we figure out a way to 
monitor this set-up.

Thanks,
Sumit
P.S. We are looking at Command Central as our long-term monitoring solution but 
need a solution for the short-term.


CONFIDENTIALITY NOTICE: This e-mail message including attachments, if any, is 
intended for the person or entity to which it is addressed and may contain 
confidential, privileged, and/or proprietary material. Any unauthorized review, 
use, disclosure or distribution is prohibited.  If you are not the intended 
recipient, please contact the sender by reply e-mail and destroy all copies of 
the original message.

Reply via email to