Thanks Guozhang for the suggestion. We are using kafka 2.3.0 and the app.id is set to the same value.
Bouncing off instances work for a small period of time but when auto scaling happens it skews it up again. This is becoming a big bottleneck for us since only one node is now trying to handle more than 60% of traffic and we are getting into huge lag. Regards, Navneeth On Mon, Dec 14, 2020 at 2:49 PM Guozhang Wang <wangg...@gmail.com> wrote: > Kafka Streams should evenly distribute the partitions, but there are some > issues in old versions of Kafka that you may be observing. To verify if it > is a transient issue or it is permanently, I'd suggest you try: > > 1) bounce the instances that have no partitions assigned (not bounce them > all together, but in rolling manner), and see if after the rebalance they > are evenly distributed. > > 2) check if the app.id are set to the same value. > > Also which version of Kafka are you using? > > > Guozhang > > > > > On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan < > reachnavnee...@gmail.com> > wrote: > > > Hi All, > > > > How does kafka streams partition assignment work for sources? I have a > > stream application reading from a topic which has 24 partitions. There > are > > 6 application containers with 4 stream tasks in each container running > but > > only 2 instances are assigned with partitions and even within the two, > one > > has 21 partitions and the other has 3 partitions assigned. There is > > something wrong in the way the partitions are assigned. Is there a way to > > evenly distribute partitions? > > > > *Group Assignment:* > > userapp user-data 5 1716495057 1716504638 > > 9581 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > > userapp user-data 13 1735721509 1735730754 > > 9245 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > > userapp user-data 9 1729975595 1729984597 > > 9002 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > > userapp user-data 17 1736457935 1736466997 > > 9062 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > > userapp user-data 0 1917953948 1917963103 > > 9155 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > > userapp user-data 11 1719082482 1719085979 > > 3497 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 19 1755067810 1755071386 > > 3576 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 7 1729598367 1729601982 > > 3615 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 1 1949943357 1949947252 > > 3895 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 15 1751889619 1751893116 > > 3497 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 3 1752352594 1752356255 > > 3661 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > > userapp user-data 6 1693612741 1693621514 > > 8773 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > > userapp user-data 2 1923482321 1923490826 > > 8505 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > > userapp user-data 14 1677960791 1677969039 > > 8248 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > > userapp user-data 10 1722133802 1722142388 > > 8586 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > > userapp user-data 18 1739152927 1739161360 > > 8433 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > > userapp user-data 20 1715428183 1715431991 > > 3808 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > > userapp user-data 8 1738854860 1738858541 > > 3681 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > > userapp user-data 4 1704061352 1704065142 > > 3790 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > > userapp user-data 12 1743370130 1743373727 > > 3597 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > > userapp user-data 16 1696027581 1696031399 > > 3818 > > > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > > /172.21.82.140 > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > > userapp user-data 21 1722643635 1722647766 > > 4131 > > > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5 > > /172.21.64.86 > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer > > userapp user-data 23 1675792619 1675800530 > > 7911 > > > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8 > > /172.21.64.86 > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer > > userapp user-data 22 1735482901 1735490720 > > 7819 > > > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2 > > /172.21.64.86 > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer > > > > Regards, > > Navneeth > > > > > -- > -- Guozhang >
