[
https://issues.apache.org/jira/browse/KAFKA-20295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101141#comment-18101141
]
Yunseop Eom commented on KAFKA-20295:
-------------------------------------
Opened PR https://github.com/apache/kafka/pull/23038. The fix makes controller
feature validation use the current Raft quorum voter IDs, so stale
registrations from removed dynamic-quorum voters no longer block
metadata.version upgrades. It also requires registrations for current voters
and retains the static quorum fallback for existing describers. Regression
coverage covers removed registrations and missing active registrations.
Verification passed: full :metadata:test, KafkaRaftClientReconfigTest,
metadata/raft Spotless, Checkstyle, SpotBugs, Kafka public API checker, and git
diff --check.
> Removed controllers still in metadata, blocking finalizing upgrade to 4.2.0
> ---------------------------------------------------------------------------
>
> Key: KAFKA-20295
> URL: https://issues.apache.org/jira/browse/KAFKA-20295
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Environment: Kafka 4.2.0 (Scala 2.13) running on Debian Trixie 13.3
> Reporter: Roland Sommer
> Priority: Major
>
> While upgrading our kafka clusters to new operating systems I switched to
> dynamic voter configuration and removed controller instances with
> {{/opt/kafka/bin/kafka-metadata-quorum.sh}} and the {{remove-controller}}
> subcommand. Inspecting the cluster with {{describe}} only shows the actual
> running nodes.
> Now during the update to 4.2.0, the final metadata upgrade step complains
> about
> {code:java}
> Could not upgrade eligible.leader.replicas.version to 1. The update failed
> for all features since the following feature had an error: Invalid update
> version 29 for feature metadata.version. Controller 351 only supports
> versions 7-27{code}
> with 351 being an ID of an already removed controller. Inspecting a snapshot
> with {{/opt/kafka/bin/kafka-metadata-shell.sh}} indeed shows all controller
> ids of already removed controllers:
> {code:java}
> >> ls image/cluster/controllers/
> 158 206 351 584 611 686 {code}
> while other tools only show the expected nodes:
> {code:java}
> ~$ /opt/kafka/bin/kafka-metadata-quorum.sh --bootstrap-controller
> localhost:9093 describe --replication --human-readable
> NodeId DirectoryId LogEndOffset Lag LastFetchTimestamp LastCaughtUpTimestamp
> Status
> 158 2gsvOvnT7urpZcA_-LUy5w 196823524 0 7 ms ago 8 ms ago Leader
> 611 27Ii-xdAZ7ReQBLsvvJb0A 196823524 0 348 ms ago 348 ms ago Follower
> 206 Q7X9o3XbKxk_3tz4T8torg 196823524 0 348 ms ago 348 ms ago Follower
> 226 7n6aedUEuytkqhBnbe7ESw 196823524 0 348 ms ago 348 ms ago Observer
> 181 tZ17VQ8cYpf7R-LyAQWf2w 196823524 0 349 ms ago 349 ms ago Observer
> 299 P4qXt3K0G5Qg_7w_UdvaNA 196823524 0 348 ms ago 348 ms ago Observer
> 290 bA0pqZFsUa45lRTB6bS4bg 196823524 0 348 ms ago 348 ms ago Observer
> 293 Av_12222lURKVYVt-aNKOQ 196823524 0 348 ms ago 348 ms ago Observer
> 485 glENIgkIng1MYDF8HxxoDQ 196823524 0 349 ms ago 350 ms ago Observer {code}
> Grepping through {{bin/kafka-dump-log.sh --cluster-metadata-decoder}} only
> shows the expected three {{REGISTER_CONTROLLER_RECORD}} entries.
> Is there any clear path for removing those stale nodes?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
