Hi Fares, You're right in your description of the contents of the __consumer_offsets topic, and how they are stored. The most common reason for an uneven load on the consumer offsets are.
1. Configuration of offset commits in the client 2. Load on topic being consumed If a topic has 10 partitions, and the producer produces records with a key and a partitioner based on the key hash, then it can happen that one or more partitions get more records than the others. Just because several keys are more often used than others. Now the consumer needs to read more records from those partitions. If the consumer commits offsets at a time interval or after an N amount of records consumed, it follows that this results in more offset commits for the topic partitions containing more records. You might want to check the load on the topic partitions being consumed to confirm this is the case. Unfortunately, I do not have an easy answer on how to remedy that problem. You can check what the offset commit settings are for your application, and if you can update the logic to reflect the higher load. If the trigger is based on time or nrOfRecords consumed you can alter these values to make sure they aren't triggered that often. You can also start monitoring and updating the cluster with a preferred leader election setting for your partitions using CruiseControl to minimize the load on your brokers. I do recommend to use keys with the hash based partitioner only if the order of messages for those keys MUST be guaranteed, else you can use a different Partitioner. Then there will be a more uniform distribution of records on the topic, and a better distribution of offset commits. Do note that there is an issue with the UniformStickyPartitioner that is being worked on right now, see https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner I hope this helps Kind regards, Richard Bosch Developer Advocate Axual BV E : richard.bo...@axual.com M : +31 6 11 850 846 W : www.axual.com On Thu, Apr 27, 2023 at 4:19 PM Fares Oueslati <oueslati.fa...@gmail.com> wrote: > Hello Kafka users, > > I’m facing an issue with a Kafka cluster, specifically with the > __consumer_offsets topic. > There seems to be an imbalance in the number of commit messages across its > partitions. Most of the commit messages are concentrated in a single > partition, which is causing high CPU usage on the broker handling that > partition. > I have already verified that the topic partitions’ leaders are > well-balanced across the six brokers. > However, a specific consumer group (The largest one, with many members > consuming from multiple topics, based on Spring Kafka) generates a large > number of commit messages, and they all end up in the same partition #37. > My understanding is that, by default, all commit messages sent by a > particular consumer group for a specific topic partition are directed to a > single partition of the __consumer_offsets topic, determined by hashing the > consumer group id and the topic partition. In our case, this default > partitioning strategy seems to be causing the imbalance, even though I > don’t understand why exactly. > Could you please help me understand why there’s such an imbalance in the > number of messages across the __consumer_offsets partitions and why the > large number of commit messages from the large consumer group are not > spread well across the partitions of the __consumer_offsets topic? Are > there any recommendations or best practices to address this issue? > > Any guidance would be greatly appreciated. > > Best Regards, > Fares >
