Adeetya Upadhyay created KAFKA-21491:
----------------------------------------
Summary: Controller resigns repeatedly because brokersToElrs is
not updated when replaying ClearElrRecord
Key: KAFKA-21491
URL: https://issues.apache.org/jira/browse/KAFKA-21491
Project: Kafka
Issue Type: Bug
Components: controller, kraft
Affects Versions: 4.3.0, 4.2.0, 4.1.0, 4.0.0
Reporter: Adeetya Upadhyay
Replaying a {{ClearElrRecord}} does not update {{brokersToElrs}}. The record is
written whenever {{min.insync.replicas}} changes, and {{removeTopicElrs()}}
replaces the partition registration directly rather than calling
{{updatePartitionInfo()}}, so the index keeps brokers that are no longer in any
ELR. Replaying {{RemoveTopicRecord}} only drops index entries for the
partition's current ELR, which is empty by then, so the stale entry outlives
the topic.
When that broker later restarts after an unclean shutdown, or is unregistered,
{{generateLeaderAndIsrUpdates()}} looks up the deleted topic and throws:
{noformat}
java.lang.RuntimeException: Topic ID <id> existed in isrMembers, but not in the
topics map.
at
org.apache.kafka.controller.ReplicationControlManager.generateLeaderAndIsrUpdates(ReplicationControlManager.java:2161)
{noformat}
This is not an {{ApiException}}, so the controller resigns. The next leader has
the same state and fails the same way.
h3. How to reproduce
Trunk at 2c620cd, default configuration, one KRaft controller, brokers 1, 2 and
3.
# Create topic {{foo}} with replicas 1,2,3 and {{min.insync.replicas=2}}.
# {{kill -9}} broker 3 and wait for ISR = [1,2]. Then {{kill -9}} broker 1. The
partition shows {{Isr: 2 Elr: 1}}.
# Set {{min.insync.replicas=1}} on {{foo}}, then delete {{foo}}.
# Restart broker 1, or run {{kafka-cluster.sh unregister --id 1}} while it is
down.
Broker 1 never rejoins. The controller resigns on each registration attempt,
and the leader epoch went from 1 to 61 in about a minute. Unregistering fails
with {{UnknownServerException}} and the controller resigns again. Restarting
the controller does not help unless a snapshot was written after the topic was
deleted, because log replay recreates the stale entry.
h3. Fix
Call {{brokersToElrs.update()}} from {{removeTopicElrs()}} so the index follows
the registration. A regression test is added to
{{ReplicationControlManagerTest#testElrsRemovedOnMinIsrUpdate}}.
The problem has been present since KAFKA-16540.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)