Perf tuning is always tricky... 350 rec/sec sounds pretty low though.
You would first need to figure out where the bottleneck is. Kafka
Streams exposes all kind of metrics:
https://kafka.apache.org/documentation/#kafka_streams_monitoring
Might be good to inspect them as a first step -- maybe something is off
and gives a first direction.
In general, it would be good to limit it to Kafka network I/O, local
RocksDB disk I/O, or CPU utilization -- each one could be the bottleneck
and we would need to first know which one before you can take any action
to change configurations.
HTH.
-Matthias
On 4/4/24 7:21 PM, Nirmal Das wrote:
Hi All,
My streams application is not processing more than 350 records/sec on a
high load of 3milliom records produced every 2-3 minutes.
My scenarios are as below -
I am on Kafka and streams version of 3.5.1 .
My key-value pair is in protobuf format .
I do a groupbykey followed by TimeWindow of 10 mins with grace period of 6
hours . It is then followed by a aggregate function which stores the first
and last offset of the record along with partition for that message key.
Am I doing something wrong? Am I doing something anti-pattern which is
throttling the system ? How can I improve this?
Regards,
Dev Lover