Hi Tilak, > While using the connector we observed that, for each call to the `put` method in our `SinkTask` implementation, there is some delay between calls which is in the range of [200ms to 1sec].
Yes, that makes sense! As you should have known, the sinkTask is also a Kafka consumer, which try to poll records periodically. And the poll timeout for the sinkTasks is basically based on the nextCommit time (i.e. `offset.flush.interval.ms` config, which default is 60 seconds) and some other conditions (ex: retry timeout). That is, what you see the delay for each `put` method [200ms to 1sec], is because Kafka connector is trying to poll records from topics, and then, call `put` methods with the records just received. The default 60 seconds interval, is considering the throughput and performance, as you can imagine, frequent committing the messages will have overhead to the tasks and brokers. You can try to reduce the `offset.flush.interval.ms` config to see if that improves the delay of each `put`. REF: https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L213-L234 Thank you. Luke On Mon, Dec 20, 2021 at 1:31 PM Tilak Sasmal <tilak.sasmal...@gmail.com> wrote: > Hello All, > > Can anyone please provide any feedback on the issue. > To add further information, we are using kafka version 2.7.1 > > Thanks & Regards, > Tilak Sasmal > > On Tue, Dec 14, 2021 at 11:46 AM Tilak Sasmal <tilak.sasmal...@gmail.com> > wrote: > > > Hello All, > > > > We are currently using Kafka Connect and a JMS sink connector (custom > > implementation) to send messages from a Kafka topic to an ActiveMQ queue, > > messages are produced in a single Kafka topic without any key > (round-robin > > partitioning). > > While using the connector we observed that, for each call to the `put` > > method in our `SinkTask` implementation, there is some delay between > calls > > which is in the range of [200ms to 1sec]. For our application we need to > > reduce this latency so that messages are published to ActiveMQ queue in > > near realtime, we tried to increase the number of sink tasks but we > didn't > > get any performance gain. > > > > Our setup for test: > > - A cluster with 5 Kafka brokers and 3 Zookeepers, > > - A cluster with 3 Kafka Connect members and 3 Zookeepers, > > - A JMS Sink connector with 3 Tasks consuming from one Kafka topic with > > round-robin partitioning having 16 partitions and producing to one > ActiveMQ > > queue. > > > > Both Kafka cluster and Kafka Connect cluster are co-located in the same > > region. > > The scenario implemented for the test is the following: send 100 messages > > using a single Kafka producer to one Kafka topic and measure the delay > > between each call to the put method in SinkTask implementation class (3 > > iterations). > > We have attached the logs having put method call with the count of > > messages received by the method. > > > > We have the following questions: > > - Is there any way/option in Kafka Connect to disable/reduce the batching > > on messages received in the put method in SinkTask? > > - Is there any way/option in Kafka Connect to increase the frequency of > > calls to put method in SinkTask? > > > > Thanks & Regards, > > Tilak Sasmal > > >
