Hello, We are currently building our data-pipeline using Confluent and as part of this implementation, we have written couple of Kafka Connect Sink Connectors for Azure and MS SQL server. To provide some more context, we are planning to use SQL Server's Change Data Capture feature to track changes happening on individual tables in the source database and then use JDBC connector's "QUERY" mode to fetch the changed data and push it to Kafka topics. We had to go down this route because we cannot guarantee the presence of Identity or Timestamp columns in our source tables so we are relying on the timestamp columns present in the system tables created by Change Data Capture (trans_begin_time) and use it along with "QUERY" mode in JDBC connector. However, this design requires us to run one JDBC connector instance per table.
On the consumer side, once the data is on Kafka, we have written an Azure connector to push the data from Kafka topics onto Azure Blob Storage and another one to push data to MS SQL server. Now coming to the actual question :) , the above design leads to a number of Kafka Connect workers, connectors and tasks running on our distributed cluster. We are planning to run multiple workers per host and have them share the tasks from our 3 connectors (JDBC, Azure and SQL Server). How can we monitor these connectors and tasks in a production environment? How can we make sure that workers are restarted once they are killed? How can we identify the failing tasks on different workers?Are there any best practices on monitoring Kafka Connect and managing workers/connectors/tasks? We are close to finishing our development and deploying this to test and production environments and it is very important that we figure out a way to monitor this set-up. Thanks, Sumit P.S. We are looking at Command Central as our long-term monitoring solution but need a solution for the short-term. CONFIDENTIALITY NOTICE: This e-mail message including attachments, if any, is intended for the person or entity to which it is addressed and may contain confidential, privileged, and/or proprietary material. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.