Hello All, We are currently using Kafka Connect and a JMS sink connector (custom implementation) to send messages from a Kafka topic to an ActiveMQ queue, messages are produced in a single Kafka topic without any key (round-robin partitioning). While using the connector we observed that, for each call to the `put` method in our `SinkTask` implementation, there is some delay between calls which is in the range of [200ms to 1sec]. For our application we need to reduce this latency so that messages are published to ActiveMQ queue in near realtime, we tried to increase the number of sink tasks but we didn't get any performance gain.
Our setup for test: - A cluster with 5 Kafka brokers and 3 Zookeepers, - A cluster with 3 Kafka Connect members and 3 Zookeepers, - A JMS Sink connector with 3 Tasks consuming from one Kafka topic with round-robin partitioning having 16 partitions and producing to one ActiveMQ queue. Both Kafka cluster and Kafka Connect cluster are co-located in the same region. The scenario implemented for the test is the following: send 100 messages using a single Kafka producer to one Kafka topic and measure the delay between each call to the put method in SinkTask implementation class (3 iterations). We have attached the logs having put method call with the count of messages received by the method. We have the following questions: - Is there any way/option in Kafka Connect to disable/reduce the batching on messages received in the put method in SinkTask? - Is there any way/option in Kafka Connect to increase the frequency of calls to put method in SinkTask? Thanks & Regards, Tilak Sasmal
logs_put_method_call.csv
Description: MS-Excel spreadsheet
