kuper created KAFKA-20839:
-----------------------------
Summary: KRaft controller stuck at MINIMUM_KRAFT_VERSION after
losing leadership race during activation
Key: KAFKA-20839
URL: https://issues.apache.org/jira/browse/KAFKA-20839
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 3.9.2
Reporter: kuper
Description
When a KRaft controller is activated on a non-empty metadata log that lacks a
FeatureLevelRecord for metadata.version, the
ActivationRecordsGenerator.recordsForNonEmptyLog() method only logs a warning
but does not write the missing record. This causes FeatureControlManager to
fall back to MINIMUM_KRAFT_VERSION (3.0-IV1), and the cluster's
metadata.version remains permanently stuck at this value.
Root Cause
This happens when the controller loses a leadership race during initial
bootstrap activation:
1. Controller formats metadata log with metadata.version = 3.9-IV0 (from
BootstrapMetadata)
2. On first activation, recordsForEmptyLog() is called and writes bootstrap
records (including the FeatureLevelRecord for metadata.version)
3. If the controller loses leadership mid-write, some records are committed but
the FeatureLevelRecord may be lost
4. When the controller regains leadership, the log is non-empty but lacks the
FeatureLevelRecord — recordsForNonEmptyLog() takes effect but previously only
logged a warning without fixing the problem
Impact
- metadata.version permanently stuck at 3.0-IV1
- KIP-919 features broken: --bootstrap-controller returns
UnsupportedVersionException: Direct-to-controller communication current
MetadataVersion
- Dynamic controller registration unavailable
- All features gated behind metadata.version >= 3.7-IV0 are blocked
Reproduction
1. Deploy a fresh KRaft 3.9.x cluster
2. During initial controller activation, trigger a leadership re-election
(e.g., network partition)
3. Once the controller stabilizes, check metadata.version:
kafka-features.sh --bootstrap-server <broker:9092> describe
4. metadata.version shows 3.0-IV1 instead of 3.9-IV0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
