Hi Jun, Thanks for the replies and questions.
RE JR1: Updated the KIP with the record schema for ClusterIdRecord. One thing I'm not sure about yet is whether or not the record field should be of UUID or String type. This is because kafka's quickstart docs refer to setting `--cluster-id` to a UUID in the storage tool. However, many places in kafka broker/controller code (e.g. the raft client, broker lifecycle manager, and even the formatter itself) only require this type to be a String. Since not all Strings are valid UUIDs, making this record field of type UUID might be too restrictive and complicate upgrading the MV for existing clusters, since they might have a non-UUID cluster id string, but need to write this record when upgrading to an MV that supports this feature. Let me know what you think. RE JR2: Any controller node formatted with `--standalone, --initial-controllers` or who is part of the static voter set defined by `controller.quorum.voters` can write the ClusterIdRecord by including the `--cluster-id` argument to `kafka-storage format`. However, if the MV of the cluster supports it, there is exactly one writer of this record to the cluster metadata partition. The writer is the first active controller, who writes this record alongside other bootstrap metadata records (e.g. metadata version) during controller activation. At this point, we already depend on MV existing, since the active controller writes these bootstrap metadata records as a transaction if the MV supports it. I think writing the cluster id record would follow a similar pattern. RE JR3: When a node formats, it will write the meta.properties file. During formatting, a node must resolve the MV it wants to format with, which is explained more in RE JR5. I need to think about this more, but I think we should keep `--cluster-id` as a required flag for invoking the format command. If a broker/observer controller does not format, meta.properties is written without cluster id immediately after startup (i.e. where we read it from disk now in KafkaRaftServer). RE JR4: Yeah, will do. In this context, when I say observers I'm referring to any controllers who are not part of the KRaft voter set when they start kafka, or any brokers. I will make this explicit in the KIP. From the perspective of this feature and KRaft leader election, controller nodes who format with `--no-initial-controllers`, controller nodes who are not part of `controller.quorum.voters`, and brokers, all do not "need" to format, since they cannot become the active controller. This means they can resolve metadata like the cluster id after discovering the leader. We have a similar pattern with how controller nodes who format with `--no-initial-controllers` discover the kraft version of the cluster. RE JR5: If a node formats, it must resolve a metadata version with which to format. This comes from the `--release-version/--feature` flag and defaults to the latest production MV. Therefore, when a node formats with a metadata version that supports this feature, it will write the ClusterIdRecord to its `0-0/bootstrap.checkpoint`. If the node formats with a metadata version that does not support this feature, it does not write ClusterIdRecord to its `0-0/bootstrap.checkpoint`. If a node skips formatting, it is assumed that this node is part of a cluster whose MV supports this. Otherwise, this is a misconfiguration and the node will fail to register with the leader since there is no way for it to persist cluster id to its meta.properties without formatting. Although I did not specify this yet on the KIP explicitly, after some offline discussion I think it makes sense to enforce the following invariant as part of the feature design: if the persisted metadata version supports this feature, the ClusterId record must also be persisted. This is enforceable on the write-path for MV, which occurs at two points-- during formatting and during feature upgrades. There is a similar pattern with kraft.version, as it gets written to disk at the same two points. RE JR6: The main motivation for writing cluster id to meta.properties as well is because it can act as a projection of the cluster metadata partition which essentially only exposes the cluster id to readers. For example, the raft layer needs to be aware of the cluster id for its own RPC handling/validation, but raft cannot read metadata records. There are many readers of this cluster id value during the startup of the cluster. Therefore, avoiding a read of the metadata partition to discover the value of this metadata will prevent more complications of the startup code. Best, Kevin Wu On Tue, Feb 17, 2026 at 7:35 PM Jun Rao via dev <[email protected]> wrote: > Hi, Kevin, > > Thanks for the KIP. A few comments. > > JR1. ClusterIdRecord : Could you define the record format? > > JR2. "a new MetadataVersion that supports encoding/decoding this record. > This means that during formatting, the bootstrap ClusterIdRecord is only > written if the cluster is formatted with a MV that supports this feature." > Could you describe who writes the ClusterIdRecord? Is it the leader > controller? Also, when is the record written? Do we guarantee that MV is > available at that time? > > JR3. "meta.properties can be written during kafka broker/controller startup > if it doesn't exist already (from formatting)" > Could you describe when meta.properties is written? Is MV available at that > time? > > JR4. "Introduce a metadata record for cluster id + observers persist > cluster id to meta.properties from metadata publishing pipeline" > Could you clarify what observers are? Are they observer controllers or are > they brokers (which are referred to as observers to the controller)? > > JR5. "Bootstrap controllers can add a mandatory “cluster id” record during > formatting" > This sounds like adding a ClusterIdRecord is optional. If so, could you > describe when a record will be added and when a record will not be added? > > JR6. "However, kafka should still be able to handle the case where a leader > is elected without a cluster id in meta.properties , since KRaft does not > need cluster.id in order to elect a leader. > In this case, the active controller will write a cluster id > record during the bootstrap metadata write." > Hmm, earlier, the KIP says "Upon discovering the cluster ID for the first > time, these nodes need to persist this to meta.properties". Why do we need > to introduce a separate place to write the cluster id to meta.properties. > > Jun > > > On Wed, Feb 11, 2026 at 10:21 AM Kevin Wu <[email protected]> wrote: > > > Hi all, > > > > Manually bumping this thread after finalizing a design. > > KIP link: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories > > > > Best, > > Kevin Wu > > > > On Tue, Jan 6, 2026 at 7:18 AM Kevin Wu <[email protected]> wrote: > > > > > Hello all, > > > > > > I would like to start a discussion on KIP-1262, which proposes removing > > > the formatting requirement for brokers and observer controllers. > > Currently, > > > I am considering two high-level designs, and would appreciate community > > > feedback on both approaches to decide on a final design. > > > > > > KIP link: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1262%3A+Enable+auto-formatting+directories > > > > > > Best, > > > Kevin Wu > > > > > >
