Hi José, RE JS1: I have updated the KIP to discuss the formatting and KRaft configuration requirements on a node level rather than a cluster level.
> Do you mean that to establish a KRaft leader, controller nodes must be formatted with "kafka-storage format ... --standalone" or "kafka-storage format ... --initial-controller" for a dynamic controller cluster? And that the kraft controllers must specify the controller.quorum.voters properties for a static controller quorum? Yes, this is what I mean. RE JS2: Sure, I think we can add this as part of this KIP, since it seems straightforward. Upon discovery of the clusterId from the metadata, the local node will upgrade any MPV0 or MPV1 to MPV2, and persist that to disk. RE JS3: I have added this to the KIP. Thanks for the discussions, Kevin Wu On Tue, Sep 22, 2026 at 12:22 PM José Armando García Sancio via dev < [email protected]> wrote: > Thanks for the responses. > > RE JS1: > > I am still having a hardtime understanding what we mean by cluster > when we are discussing kafka-storage format. Kafka-storage format is a > node operation not a cluster-wide operation. > > In your response, you have "In this case, `controller.quorum.voters` > must be present to elect a KRaft leader. `controller.quorum.voters` is > not required on nodes who skip formatting." Why not just say that? > When explaining a distributed system I find it helpful to define the > protocol in terms of individual nodes since the code implementing the > protocol executes on individual nodes. > > To me the statement "Clusters who skip formatting will also have a > static KRaft quorum, which is explained in the below section, so > controller.quorum.voters must be defined. Operators can still end up > with a dynamic quorum without formatting because static to dynamic > quorum upgrades are supported." is not really implementable. > > Do you mean that to establish a KRaft leader, controller nodes must be > formatted with "kafka-storage format ... --standalone" or > "kafka-storage format ... --initial-controller" for a dynamic > controller cluster? And that the kraft controllers must specify the > controller.quorum.voters properties for a static controller quorum? > > > RE JS2: > "For clusters with MV >= 150, it is fine for > a given node to have MPV1 because it was previously formatted (this > includes all existing clusters), but if a given node was not formatted, it > will have MPV2." > > It is fine for the node to be in meta.properties version 1 but the > question is whether we should upgrade the meta.properties version to 2 > if the MV is upgraded to 150 in your example. It is good practice to > limit the use of older versions so that eventually we can deprecate > them. > > RE JS3: > > Sounds good. > > On Mon, Sep 21, 2026 at 3:00 PM Kevin Wu <[email protected]> wrote: > > > > Hi José, > > > > Thanks for the questions. > > > > RE JS1: > > I think my language may have been a bit confusing. What I mean by > "clusters > > who skip formatting" is a cluster in which no node (controller or broker) > > invokes `kafka-storage format` before starting kafka. In this case, > > `controller.quorum.voters` must be present to elect a KRaft leader. > > `controller.quorum.voters` is not required on nodes who skip formatting. > > > > For example, to startup a new KIP-853 cluster with auto-formatting, the > > operator could format one node with `--standalone`, and then skip > > formatting on all other nodes. If the operator wants to add a new > observer > > controller to this existing KIP-853 cluster, the operator can also skip > > formatting that node, and that node does not need > > `controller.quorum.voters` so long as > `controller.quorum.bootstrap.servers` > > is defined to discover the quorum. I explained some of these things in > this > > section section: > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-Whenisformattingstillrequiredbeforestartingkafka__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACZKHDG7G$ > . > > I can add some of the discussion here to that section of the KIP. Does > this > > answer your questions? > > > > For now, skipping formatting entirely for a new KIP-853 cluster is out of > > scope. See RE JR12 for details as to why. > > > > RE JS2: > > I am not sure if I agree that a cluster's MV determines which version of > > meta.properties to write. I believe only one thing should determine the > > meta.properties version (MPV), which is whether the operator formatted > the > > local node. Assume the MV this feature introduces is MV 150. The > > auto-formatting feature works with both MPV1 and MPV2. The MPV1 case is > > less interesting because it means the clusterId is already known at > format > > time. > > > > When the operator invokes `kafka-storage format`, MPV 1 is > > written, regardless of the MV. > > > > On the other hand, if the operator skips formatting on any node, the only > > valid cluster MVs are necessarily the ones that support > `ClusterIdRecord`, > > i.e. MV >= 150. For clusters with MV < 150, brokers and controllers that > > skip formatting will fail to complete startup, since there is no way for > > them to discover the clusterId, and ideally, we should report this > > misconfiguration in the error. For clusters with MV >= 150, it is fine > for > > a given node to have MPV1 because it was previously formatted (this > > includes all existing clusters), but if a given node was not formatted, > it > > will have MPV2. > > > > Because this feature is gated behind a finalized MV 150, we do not have > to > > worry about MPV backwards compatibility, because it is not a valid > > operation to downgrade the software version of a cluster with a finalized > > MV 150 to a software version that does not support MV 150. > > > > RE JS3: > > The high level idea was to block broker startup on discovering the > > clusterId, since broker registration, which also blocks startup and > > requires the broker to catch up with the metadata, relies on the > clusterId. > > On the broker, this means Kafka will wait for ` > server.max.startup.time.ms` > > to discover the clusterId. I can include more details about this in this > > section: > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-CompletionofBrokerServerandControllerServerstartupmustblockonthediscoveryandpersistenceofthecluster.idtometa.properties__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACbJaNL6S$ > > > > On the controller, the controller also waits for ` > server.max.startup.time.ms` > > to finish startup. I believe it is fine to wait for the clusterId to be > > committed after the existing blocking for authorizer futures to complete. > > > > Best, > > Kevin Wu > > > > On Mon, Sep 21, 2026 at 12:48 PM José Armando García Sancio via dev < > > [email protected]> wrote: > > > > > Hi Kevin, > > > > > > I was able to take a look at the KIP today. > > > > > > JS1: > > > > > > In the > > > > Clusters who skip formatting will also have a static KRaft quorum, > which > > > is explained in the below section, so controller.quorum.voters must be > > > defined. Operators can still end up with a dynamic quorum without > > > formatting because static to dynamic quorum upgrades are supported. > > > > > > What do you mean by "Clusters who skip formatting"? Clusters are not > > > formatted. Users format nodes instead. Why is controller.quorum.voters > > > required if a node skips formatting? If this is a requirement, how > > > would the user provision a new controller (observer) node to join an > > > existing cluster through KIP-853? The controller.quorum.voters > > > properties should not be required in a node that will join a dynamic > > > controller cluster. > > > > > > JS2: > > > > > > When to write version 2 of the meta.properties file? I am trying to > > > understand when Kafka will write version 2 of the meta.properties > > > file. I would assume that this is based on the MV but I don't see this > > > stated explicitly in the KIP. When the MV changes to a version that > > > supports meta.properties version 2 will Kafka automatically upgrade > > > the version even if there are no changes to the state stored in > > > meta.properties? > > > > > > JS3: > > > > > > How long will Kafka wait for the cluster id to be discovered? What > > > happens if the cluster id is not in the cluster metadata partition and > > > all of the meta.properties files? > > > > > > Thanks, > > > -- > > > -José > > > > > > > -- > -José >
