Hello All, Can anyone please provide any feedback on the issue. To add further information, we are using kafka version 2.7.1
Thanks & Regards, Tilak Sasmal On Tue, Dec 14, 2021 at 11:46 AM Tilak Sasmal <tilak.sasmal...@gmail.com> wrote: > Hello All, > > We are currently using Kafka Connect and a JMS sink connector (custom > implementation) to send messages from a Kafka topic to an ActiveMQ queue, > messages are produced in a single Kafka topic without any key (round-robin > partitioning). > While using the connector we observed that, for each call to the `put` > method in our `SinkTask` implementation, there is some delay between calls > which is in the range of [200ms to 1sec]. For our application we need to > reduce this latency so that messages are published to ActiveMQ queue in > near realtime, we tried to increase the number of sink tasks but we didn't > get any performance gain. > > Our setup for test: > - A cluster with 5 Kafka brokers and 3 Zookeepers, > - A cluster with 3 Kafka Connect members and 3 Zookeepers, > - A JMS Sink connector with 3 Tasks consuming from one Kafka topic with > round-robin partitioning having 16 partitions and producing to one ActiveMQ > queue. > > Both Kafka cluster and Kafka Connect cluster are co-located in the same > region. > The scenario implemented for the test is the following: send 100 messages > using a single Kafka producer to one Kafka topic and measure the delay > between each call to the put method in SinkTask implementation class (3 > iterations). > We have attached the logs having put method call with the count of > messages received by the method. > > We have the following questions: > - Is there any way/option in Kafka Connect to disable/reduce the batching > on messages received in the put method in SinkTask? > - Is there any way/option in Kafka Connect to increase the frequency of > calls to put method in SinkTask? > > Thanks & Regards, > Tilak Sasmal >
