[
https://issues.apache.org/jira/browse/KAFKA-12405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KAFKA-12405:
------------------------------
Description:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) All topics with same
configuration:
Topic:xxx.metrics PartitionCount:1 ReplicationFactor:3 Configs:
Topic: xxx.metrics Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 +0000 [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 +0000 [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 +0000 [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
was:
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to "Out of sync replicas"
* Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
2021-03-02 11:34:38,791 +0000 [executor-Produce]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]: Preparing
to rebalance group mirror_maker_group in state PreparingRebalance with old
generation 7139 (__consumer_offsets-30) (reason: error when storing group
assignment during SyncGroup (member:
mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,837 +0000 [data-plane-kafka-request-handler-6]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
(kafka.coordinator.group.GroupCoordinator)
2021-03-02 11:34:38,885 +0000 [data-plane-kafka-request-handler-7]
[kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
Assignment received from leader for group mirror_maker_group for generation
7140 (kafka.coordinator.group.GroupCoordinator){noformat}
The number of log entry "Rebalancing group..." per hour
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, was this fixed in any later kafka release?
Thank you.
> Inconsistent broker without active topics still involved in coordinating
> consumer group
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-12405
> URL: https://issues.apache.org/jira/browse/KAFKA-12405
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.3.0
> Reporter: Alexander
> Priority: Major
>
> Short description:
> Broker without active topics and with inconsistent data still involved in
> coordinating consumer group. It leads to constant rebalancing consumer group
>
> Details:
> We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3) All topics with
> same configuration:
> Topic:xxx.metrics PartitionCount:1 ReplicationFactor:3 Configs:
> Topic: xxx.metrics Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2
> * After high disk load broker(2) came out of cluster
> * Brokers (1), (3) shortly became leaders of it's topics
> * Broker(2) back to cluster, but have failed to became leader of it's
> preferred topics due to "Out of sync replicas"
> * Now it start spamming with
> "Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
> "Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
> Like in https://issues.apache.org/jira/browse/KAFKA-3042
> But at the same time we start to observing constant rebalancing of consumer
> group "mirror_maker_group" and topic reading within this group was completely
> stucked. Restarting broker(2) fix the problem
> Broker(2) server log:
> {noformat}
> 2021-03-02 11:34:38,791 +0000 [executor-Produce]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Preparing to rebalance group mirror_maker_group in state PreparingRebalance
> with old generation 7139 (__consumer_offsets-30) (reason: error when storing
> group assignment during SyncGroup (member:
> mirror_maker_group-0-0668b084-bef9-46f6-aec2-07807f1634a0))
> (kafka.coordinator.group.GroupCoordinator)
> 2021-03-02 11:34:38,837 +0000 [data-plane-kafka-request-handler-6]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Stabilized group mirror_maker_group generation 7140 (__consumer_offsets-30)
> (kafka.coordinator.group.GroupCoordinator)
> 2021-03-02 11:34:38,885 +0000 [data-plane-kafka-request-handler-7]
> [kafka.coordinator.group.GroupCoordinator] INFO [GroupCoordinator 2]:
> Assignment received from leader for group mirror_maker_group for generation
> 7140 (kafka.coordinator.group.GroupCoordinator){noformat}
> The number of log entry "Rebalancing group..." per hour
> {noformat}
> $ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`;
> do echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group'
> $log | wc -l; done
> server.2021-03-02-10.kafka3 5 # kafka3 - fine node with actvice controller
> server.2021-03-02-11.kafka3 1
> server.2021-03-02-12.kafka3 3
> server.2021-03-02-10.kafka2 1 # kafka2 - node with inconsistent broker
> server.2021-03-02-11.kafka2 306
> server.2021-03-02-12.kafka2 452
> server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
> {noformat}
>
> The questions:
> * Is it a bug or topicless broker should be involved in coordinating
> consumer group?
> * If it's a bug, was this fixed in any later kafka release?
> Thank you.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
