Hi, Lee, Would you be able to see which Kafka API is generating the traffic? This is provided by the MBean kafka.network:type=RequestMetrics,name=RequestsPerSec,request=*,version=([0-9]+) [1].
Thanks, Alexandre [1] https://kafka.apache.org/documentation/#monitoring Le dim. 16 avr. 2023 à 18:22, Lee Carroll <lee.a.carr...@googlemail.com.invalid> a écrit : > > Hi Kirk, > > Points 1 and 2: yes we have both logging and metrics. I'll collate > Point 3 concurrency refers to the spring boot kafka concurrent container. 2 > threads will be made available to the consumer group to consume 30 > partitions making up a single topic. > point 4: consumer- Kafka Consumer Configurations for Confluent Platform | > Confluent Documentation > <https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms> > > On Thu, 13 Apr 2023 at 16:03, Kirk True <k...@kirktrue.pro> wrote: > > > Hi Lee, > > > > Some questions: > > > > 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links > > on consumer-level and fetch-level metrics, respectively. > > 2. Can you enable more detailed logging? > > 3. What do you mean specifically by “concurrency of 2?” > > 4. Can you verify the name of the fetch.wait.max.ms configuration? I > > don’t see that configuration option. Is that a broker configuration or a > > consumer configuration? > > > > Thanks, > > Kirk > > > > [1] https://kafka.apache.org/documentation/#consumer_group_monitoring > > [2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring > > > > > On Apr 13, 2023, at 4:52 AM, Lee Carroll > > > <lee.a.carr...@googlemail.com.INVALID> > > wrote: > > > > > > Hi All, > > > I've a spring boot kafka consumer group, consuming a topic with 30 > > > partitions with a concurrency of 2 (staging set up). > > > > > > The network traffic in and out was very high >200k per second even when > > > consuming no messages. > > > > > > I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than > > > the default which has had a significant impact but the network is still > > > seeing ~30k a second. > > > > > > Is this to be expected? > > > > > > Cheers Lee C > > > >