Hi Kirk, Points 1 and 2: yes we have both logging and metrics. I'll collate Point 3 concurrency refers to the spring boot kafka concurrent container. 2 threads will be made available to the consumer group to consume 30 partitions making up a single topic. point 4: consumer- Kafka Consumer Configurations for Confluent Platform | Confluent Documentation <https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#fetch-max-wait-ms>
On Thu, 13 Apr 2023 at 16:03, Kirk True <k...@kirktrue.pro> wrote: > Hi Lee, > > Some questions: > > 1. Can you enable metrics for your consumer(s)? See [1] and [2] for links > on consumer-level and fetch-level metrics, respectively. > 2. Can you enable more detailed logging? > 3. What do you mean specifically by “concurrency of 2?” > 4. Can you verify the name of the fetch.wait.max.ms configuration? I > don’t see that configuration option. Is that a broker configuration or a > consumer configuration? > > Thanks, > Kirk > > [1] https://kafka.apache.org/documentation/#consumer_group_monitoring > [2] https://kafka.apache.org/documentation/#consumer_fetch_monitoring > > > On Apr 13, 2023, at 4:52 AM, Lee Carroll > > <lee.a.carr...@googlemail.com.INVALID> > wrote: > > > > Hi All, > > I've a spring boot kafka consumer group, consuming a topic with 30 > > partitions with a concurrency of 2 (staging set up). > > > > The network traffic in and out was very high >200k per second even when > > consuming no messages. > > > > I've adjusted fetch.wait.max.ms to be significantly higher (15,000) than > > the default which has had a significant impact but the network is still > > seeing ~30k a second. > > > > Is this to be expected? > > > > Cheers Lee C > >
