Adeetya Upadhyay created KAFKA-21491:
----------------------------------------

             Summary: Controller resigns repeatedly because brokersToElrs is 
not updated when replaying ClearElrRecord
                 Key: KAFKA-21491
                 URL: https://issues.apache.org/jira/browse/KAFKA-21491
             Project: Kafka
          Issue Type: Bug
          Components: controller, kraft
    Affects Versions: 4.3.0, 4.2.0, 4.1.0, 4.0.0
            Reporter: Adeetya Upadhyay


Replaying a {{ClearElrRecord}} does not update {{brokersToElrs}}. The record is 
written whenever {{min.insync.replicas}} changes, and {{removeTopicElrs()}} 
replaces the partition registration directly rather than calling 
{{updatePartitionInfo()}}, so the index keeps brokers that are no longer in any 
ELR. Replaying {{RemoveTopicRecord}} only drops index entries for the 
partition's current ELR, which is empty by then, so the stale entry outlives 
the topic.

When that broker later restarts after an unclean shutdown, or is unregistered, 
{{generateLeaderAndIsrUpdates()}} looks up the deleted topic and throws:
{noformat}
java.lang.RuntimeException: Topic ID <id> existed in isrMembers, but not in the 
topics map.
    at 
org.apache.kafka.controller.ReplicationControlManager.generateLeaderAndIsrUpdates(ReplicationControlManager.java:2161)
{noformat}
This is not an {{ApiException}}, so the controller resigns. The next leader has 
the same state and fails the same way.

h3. How to reproduce
Trunk at 2c620cd, default configuration, one KRaft controller, brokers 1, 2 and 
3.
# Create topic {{foo}} with replicas 1,2,3 and {{min.insync.replicas=2}}.
# {{kill -9}} broker 3 and wait for ISR = [1,2]. Then {{kill -9}} broker 1. The 
partition shows {{Isr: 2  Elr: 1}}.
# Set {{min.insync.replicas=1}} on {{foo}}, then delete {{foo}}.
# Restart broker 1, or run {{kafka-cluster.sh unregister --id 1}} while it is 
down.

Broker 1 never rejoins. The controller resigns on each registration attempt, 
and the leader epoch went from 1 to 61 in about a minute. Unregistering fails 
with {{UnknownServerException}} and the controller resigns again. Restarting 
the controller does not help unless a snapshot was written after the topic was 
deleted, because log replay recreates the stale entry.

h3. Fix
Call {{brokersToElrs.update()}} from {{removeTopicElrs()}} so the index follows 
the registration. A regression test is added to 
{{ReplicationControlManagerTest#testElrsRemovedOnMinIsrUpdate}}.

The problem has been present since KAFKA-16540.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to