Kafka Streams should evenly distribute the partitions, but there are some issues in old versions of Kafka that you may be observing. To verify if it is a transient issue or it is permanently, I'd suggest you try:
1) bounce the instances that have no partitions assigned (not bounce them all together, but in rolling manner), and see if after the rebalance they are evenly distributed. 2) check if the app.id are set to the same value. Also which version of Kafka are you using? Guozhang On Mon, Dec 14, 2020 at 11:29 AM Navneeth Krishnan <reachnavnee...@gmail.com> wrote: > Hi All, > > How does kafka streams partition assignment work for sources? I have a > stream application reading from a topic which has 24 partitions. There are > 6 application containers with 4 stream tasks in each container running but > only 2 instances are assigned with partitions and even within the two, one > has 21 partitions and the other has 3 partitions assigned. There is > something wrong in the way the partitions are assigned. Is there a way to > evenly distribute partitions? > > *Group Assignment:* > userapp user-data 5 1716495057 1716504638 > 9581 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > userapp user-data 13 1735721509 1735730754 > 9245 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > userapp user-data 9 1729975595 1729984597 > 9002 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > userapp user-data 17 1736457935 1736466997 > 9062 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > userapp user-data 0 1917953948 1917963103 > 9155 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer-428625d0-da9c-44d0-b499-79e698f5add0 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-1-consumer > userapp user-data 11 1719082482 1719085979 > 3497 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 19 1755067810 1755071386 > 3576 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 7 1729598367 1729601982 > 3615 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 1 1949943357 1949947252 > 3895 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 15 1751889619 1751893116 > 3497 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 3 1752352594 1752356255 > 3661 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer-a97f7c1f-f4a1-4b39-bd80-9b8c18dfd62a > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-4-consumer > userapp user-data 6 1693612741 1693621514 > 8773 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > userapp user-data 2 1923482321 1923490826 > 8505 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > userapp user-data 14 1677960791 1677969039 > 8248 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > userapp user-data 10 1722133802 1722142388 > 8586 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > userapp user-data 18 1739152927 1739161360 > 8433 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer-59fe9d37-cd32-4a96-867b-c13e85cba398 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-2-consumer > userapp user-data 20 1715428183 1715431991 > 3808 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > userapp user-data 8 1738854860 1738858541 > 3681 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > userapp user-data 4 1704061352 1704065142 > 3790 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > userapp user-data 12 1743370130 1743373727 > 3597 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > userapp user-data 16 1696027581 1696031399 > 3818 > > > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer-21543ce6-dcd3-4bf5-992a-faf3af8c5895 > /172.21.82.140 > userapp-aa8f851b-26b9-4dad-ad3c-333caeae388d-StreamThread-3-consumer > userapp user-data 21 1722643635 1722647766 > 4131 > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer-19fdbf9f-fb65-45fb-a020-aa16e9da76c5 > /172.21.64.86 > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-1-consumer > userapp user-data 23 1675792619 1675800530 > 7911 > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer-e98dae60-89b9-44b0-a366-d533b288fcf8 > /172.21.64.86 > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-3-consumer > userapp user-data 22 1735482901 1735490720 > 7819 > > > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer-3553c1a0-def4-453d-a4fb-a6c4f08ad3f2 > /172.21.64.86 > userapp-dc160b83-62db-41ea-9266-ffefe233fa83-StreamThread-2-consumer > > Regards, > Navneeth > -- -- Guozhang
