Hi we have around 70-75 topics created on kafka cluster.
And there is a consumer pod which is running in different namespace. This
consumer pod creates separate consumer group for each of the topic. when I
check the logs of consumer pod it shows the below error:
```
GroupId: shows_elk_Group, Topic: kafka-shows connected
GroupId: movies_elk_Group, Topic: kafka-movies connected
{"level":"ERROR","timestamp":"2021-07-22T10:14:33.977Z","logger":"kafkajs","message":"[Runner]
The group is rebalancing,
re-joining","groupId":"shows_elk_Group","memberId":"cms-kafka-service-612f2458-5b05-427e-8776-0e4270b2c215","error":"The
group is rebalancing, so a rejoin is needed","retryCount":0,"retryTime":92}
{"level":"ERROR","timestamp":"2021-07-22T10:14:33.165Z","logger":"kafkajs","message":"[Runner]
The group is rebalancing,
re-joining","groupId":"movies_elk_Group","memberId":"cms-kafka-service-56cda415-2414-4054-b285-6c794edf1700","error":"The
group is rebalancing, so a rejoin is needed","retryCount":0,"retryTime":102}
```
I am getting same error as above for all other consumer group as well.Also
attaching the kafka broker logs:
```
kubectl logs my-kafka-cluster-kafka-1 -n kafka| grep shows_elk_Group
2021-07-22 03:54:18,466 INFO Loaded member
MemberMetadata(memberId=cms-kafka-service-df1038e3-d5db-4e62-885f-9f8d011ae9ff,
groupInstanceId=None, clientId=cms-kafka-service, clientHost=/10.244.2.192,
sessionTimeoutMs=30000, rebalanceTimeoutMs=60000,
supportedProtocols=List(RoundRobinAssigner), ) in group shows_elk_Group
with generation 3. (kafka.coordinator.group.GroupMetadata$)
[group-metadata-manager-0]
2021-07-22 03:54:18,471 INFO Loaded member
MemberMetadata(memberId=cms-kafka-service-df1038e3-d5db-4e62-885f-9f8d011ae9ff,
groupInstanceId=None, clientId=cms-kafka-service, clientHost=/10.244.2.192,
sessionTimeoutMs=30000, rebalanceTimeoutMs=60000,
supportedProtocols=List(RoundRobinAssigner), ) in group shows_elk_Group
with generation 4. (kafka.coordinator.group.GroupMetadata$)
[group-metadata-manager-0]
2021-07-22 03:54:18,471 INFO Loaded member
MemberMetadata(memberId=cms-kafka-service-11a6a273-9a7c-4053-b6dd-1ca27d06690f,
groupInstanceId=None, clientId=cms-kafka-service, clientHost=/10.244.2.192,
sessionTimeoutMs=30000, rebalanceTimeoutMs=60000,
supportedProtocols=List(RoundRobinAssigner), ) in group shows_elk_Group
with generation 6. (kafka.coordinator.group.GroupMetadata$)
[group-metadata-manager-0]
2021-07-22 03:54:18,475 INFO [GroupCoordinator 1]: Loading group metadata
for shows_elk_Group with generation 6
(kafka.coordinator.group.GroupCoordinator) [group-metadata-manager-0]
2021-07-22 03:54:20,791 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 6
(__consumer_offsets-3) (reason: leader
cms-kafka-service-11a6a273-9a7c-4053-b6dd-1ca27d06690f re-joining group
during Stable) (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-7]
2021-07-22 03:54:20,791 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 7 (__consumer_offsets-3) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-7]
2021-07-22 03:54:20,793 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 7. The group has 1 members,
0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-0]
2021-07-22 09:25:41,999 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 7
(__consumer_offsets-3) (reason: Adding new member
cms-kafka-service-0dab8855-ebe2-4e03-bdca-b1351e122b42 with group instance
id None) (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-0]
2021-07-22 09:25:42,182 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 8 (__consumer_offsets-3) with 2 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-5]
2021-07-22 09:25:42,184 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 8. The group has 2 members,
0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-2]
2021-07-22 09:26:37,201 INFO [GroupCoordinator 1]: Member
cms-kafka-service-11a6a273-9a7c-4053-b6dd-1ca27d06690f in group
shows_elk_Group has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 09:26:37,202 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 8
(__consumer_offsets-3) (reason: removing member
cms-kafka-service-11a6a273-9a7c-4053-b6dd-1ca27d06690f on heartbeat
expiration) (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 09:26:37,270 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 9 (__consumer_offsets-3) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-0]
2021-07-22 09:26:37,271 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 9. The group has 1 members,
0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-3]
2021-07-22 10:04:27,124 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 9
(__consumer_offsets-3) (reason: Adding new member
cms-kafka-service-9a363fe5-79a2-4dfb-9c69-4171abed14da with group instance
id None) (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-5]
2021-07-22 10:04:28,645 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 10 (__consumer_offsets-3) with 2 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-4]
2021-07-22 10:04:28,647 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 10. The group has 2
members, 0 of which arestatic. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-2]
2021-07-22 10:05:23,657 INFO [GroupCoordinator 1]: Member
cms-kafka-service-0dab8855-ebe2-4e03-bdca-b1351e122b42 in group
shows_elk_Group has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 10:05:23,657 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 10
(__consumer_offsets-3) (reason: removing member
cms-kafka-service-0dab8855-ebe2-4e03-bdca-b1351e122b42 on heartbeat
expiration) (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 10:05:23,695 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 11 (__consumer_offsets-3) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-0]
2021-07-22 10:05:23,696 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 11. The group has 1
members, 0 of which arestatic. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-7]
2021-07-22 10:13:36,957 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 11
(__consumer_offsets-3) (reason: Adding new member
cms-kafka-service-612f2458-5b05-427e-8776-0e4270b2c215 with group instance
id None) (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-4]
2021-07-22 10:13:38,950 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 12 (__consumer_offsets-3) with 2 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-4]
2021-07-22 10:13:38,952 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 12. The group has 2
members, 0 of which arestatic. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-0]
2021-07-22 10:14:33,966 INFO [GroupCoordinator 1]: Member
cms-kafka-service-9a363fe5-79a2-4dfb-9c69-4171abed14da in group
shows_elk_Group has failed, removing it from the group
(kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 10:14:33,966 INFO [GroupCoordinator 1]: Preparing to rebalance
group shows_elk_Group in state PreparingRebalance with old generation 12
(__consumer_offsets-3) (reason: removing member
cms-kafka-service-9a363fe5-79a2-4dfb-9c69-4171abed14da on heartbeat
expiration) (kafka.coordinator.group.GroupCoordinator) [executor-Heartbeat]
2021-07-22 10:14:33,983 INFO [GroupCoordinator 1]: Stabilized group
shows_elk_Group generation 13 (__consumer_offsets-3) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-3]
2021-07-22 10:14:33,985 INFO [GroupCoordinator 1]: Assignment received from
leader for group shows_elk_Group for generation 13. The group has 1
members, 0 of which arestatic. (kafka.coordinator.group.GroupCoordinator)
[data-plane-kafka-request-handler-2]
```
I don't know why my consumer groups are rebalancing. Please help me to
resolve this issue. And let me know if you require any further information.
Thanks and regards
Shreyas
