ChetanKoneru commented on PR #22191: URL: https://github.com/apache/kafka/pull/22191#issuecomment-5118220254
Thank you for working on this—this is directly relevant to a production stale-controller recovery case we have. see - https://github.com/orgs/strimzi/discussions/13001 We run a Strimzi Kafka with KRaft on AKS. After upgrading Strimzi `0.42.0 -> 0.45.2` and Kafka `3.7.1 -> 3.9.2`, the cluster is healthy (`Ready=True`) but cannot finalize the metadata upgrade from `3.7-IV4` to `3.9-IV0`. ``` Strimzi: 0.45.2 Kafka: 3.9.2 KRaft metadata state: KRaft Finalized metadata: 3.7-IV4 Requested metadata: 3.9-IV0 Active brokers: 0,1,2 Active controllers: 90,91,92 status.registeredNodeIds: [0,1,2,90,91,92] ``` During the original ZooKeeper-to-KRaft migration, the controller NodePool did not have reserved node IDs, so controllers were assigned `3,4,5`. We later reserved IDs explicitly: ``` broker NodePool strimzi.io/next-node-ids: "[0-49]" controller NodePool strimzi.io/next-node-ids: "[90-99]" ``` The active controller NodePool now uses `90,91,92`. Controllers `3,4,5` are no longer Pods or quorum voters, but their registrations remain in `__cluster_metadata`: ``` ControllerRegistration(id=3, supportedFeatures={metadata.version: 1-19}) ControllerRegistration(id=4, supportedFeatures={metadata.version: 1-19}) ControllerRegistration(id=5, supportedFeatures={metadata.version: 1-19}) ``` The active controllers and brokers support metadata feature levels `1-21`. The metadata upgrade fails because the stale registration still participates in validation: ``` InvalidUpdateVersionException: Invalid update version 21 for feature metadata.version. Controller 5 only supports versions 1-19 ``` As the safe mitigation, we retain Kafka `3.9.2` but keep the desired metadata version at `3.7-IV4`. This stops repeated failed update attempts, but leaves the expected incomplete-upgrade warning. We understand this PR adds `Admin#unregisterController`, `kafka-cluster.sh unregister-controller`, and `UnregisterControllerRecord`, which is exactly the missing capability. Could you clarify whether this operation requires the cluster to have already finalized the metadata version that supports `UnregisterControllerRecord` (currently proposed as `IBP_4_4_IV1`)? If so, this existing-cluster recovery case appears circular: 1. Stale controller registrations must be removed to advance metadata. 2. The proposed controller-unregistration record requires newer metadata. 3. Advancing metadata is blocked by those same stale registrations. Is an in-place repair path for clusters stuck at an older finalized metadata version in scope for this PR or planned separately? In particular, is there a supported way to unregister obsolete controller registrations while retaining `metadata.version=3.7-IV4`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
