Thanks for the responses. RE JS1:
I am still having a hardtime understanding what we mean by cluster when we are discussing kafka-storage format. Kafka-storage format is a node operation not a cluster-wide operation. In your response, you have "In this case, `controller.quorum.voters` must be present to elect a KRaft leader. `controller.quorum.voters` is not required on nodes who skip formatting." Why not just say that? When explaining a distributed system I find it helpful to define the protocol in terms of individual nodes since the code implementing the protocol executes on individual nodes. To me the statement "Clusters who skip formatting will also have a static KRaft quorum, which is explained in the below section, so controller.quorum.voters must be defined. Operators can still end up with a dynamic quorum without formatting because static to dynamic quorum upgrades are supported." is not really implementable. Do you mean that to establish a KRaft leader, controller nodes must be formatted with "kafka-storage format ... --standalone" or "kafka-storage format ... --initial-controller" for a dynamic controller cluster? And that the kraft controllers must specify the controller.quorum.voters properties for a static controller quorum? RE JS2: "For clusters with MV >= 150, it is fine for a given node to have MPV1 because it was previously formatted (this includes all existing clusters), but if a given node was not formatted, it will have MPV2." It is fine for the node to be in meta.properties version 1 but the question is whether we should upgrade the meta.properties version to 2 if the MV is upgraded to 150 in your example. It is good practice to limit the use of older versions so that eventually we can deprecate them. RE JS3: Sounds good. On Mon, Sep 21, 2026 at 3:00 PM Kevin Wu <[email protected]> wrote: > > Hi José, > > Thanks for the questions. > > RE JS1: > I think my language may have been a bit confusing. What I mean by "clusters > who skip formatting" is a cluster in which no node (controller or broker) > invokes `kafka-storage format` before starting kafka. In this case, > `controller.quorum.voters` must be present to elect a KRaft leader. > `controller.quorum.voters` is not required on nodes who skip formatting. > > For example, to startup a new KIP-853 cluster with auto-formatting, the > operator could format one node with `--standalone`, and then skip > formatting on all other nodes. If the operator wants to add a new observer > controller to this existing KIP-853 cluster, the operator can also skip > formatting that node, and that node does not need > `controller.quorum.voters` so long as `controller.quorum.bootstrap.servers` > is defined to discover the quorum. I explained some of these things in this > section section: > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-Whenisformattingstillrequiredbeforestartingkafka__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACZKHDG7G$ > . > I can add some of the discussion here to that section of the KIP. Does this > answer your questions? > > For now, skipping formatting entirely for a new KIP-853 cluster is out of > scope. See RE JR12 for details as to why. > > RE JS2: > I am not sure if I agree that a cluster's MV determines which version of > meta.properties to write. I believe only one thing should determine the > meta.properties version (MPV), which is whether the operator formatted the > local node. Assume the MV this feature introduces is MV 150. The > auto-formatting feature works with both MPV1 and MPV2. The MPV1 case is > less interesting because it means the clusterId is already known at format > time. > > When the operator invokes `kafka-storage format`, MPV 1 is > written, regardless of the MV. > > On the other hand, if the operator skips formatting on any node, the only > valid cluster MVs are necessarily the ones that support `ClusterIdRecord`, > i.e. MV >= 150. For clusters with MV < 150, brokers and controllers that > skip formatting will fail to complete startup, since there is no way for > them to discover the clusterId, and ideally, we should report this > misconfiguration in the error. For clusters with MV >= 150, it is fine for > a given node to have MPV1 because it was previously formatted (this > includes all existing clusters), but if a given node was not formatted, it > will have MPV2. > > Because this feature is gated behind a finalized MV 150, we do not have to > worry about MPV backwards compatibility, because it is not a valid > operation to downgrade the software version of a cluster with a finalized > MV 150 to a software version that does not support MV 150. > > RE JS3: > The high level idea was to block broker startup on discovering the > clusterId, since broker registration, which also blocks startup and > requires the broker to catch up with the metadata, relies on the clusterId. > On the broker, this means Kafka will wait for `server.max.startup.time.ms` > to discover the clusterId. I can include more details about this in this > section: > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/KAFKA/pages/406618510/KIP-1262*Enable*auto-formatting*directories*KIP1262*3AEnableautoformattingdirectories-CompletionofBrokerServerandControllerServerstartupmustblockonthediscoveryandpersistenceofthecluster.idtometa.properties__;KysrIyU!!Ayb5sqE7!rIR_OnksedyKJKLdEHtQGHmBmhnGtMq3ZIkdfj1x5p9a-6QAv2r6PvvULfHrixiKqADTcReYdd1r9QkACbJaNL6S$ > > On the controller, the controller also waits for `server.max.startup.time.ms` > to finish startup. I believe it is fine to wait for the clusterId to be > committed after the existing blocking for authorizer futures to complete. > > Best, > Kevin Wu > > On Mon, Sep 21, 2026 at 12:48 PM José Armando García Sancio via dev < > [email protected]> wrote: > > > Hi Kevin, > > > > I was able to take a look at the KIP today. > > > > JS1: > > > > In the > > > Clusters who skip formatting will also have a static KRaft quorum, which > > is explained in the below section, so controller.quorum.voters must be > > defined. Operators can still end up with a dynamic quorum without > > formatting because static to dynamic quorum upgrades are supported. > > > > What do you mean by "Clusters who skip formatting"? Clusters are not > > formatted. Users format nodes instead. Why is controller.quorum.voters > > required if a node skips formatting? If this is a requirement, how > > would the user provision a new controller (observer) node to join an > > existing cluster through KIP-853? The controller.quorum.voters > > properties should not be required in a node that will join a dynamic > > controller cluster. > > > > JS2: > > > > When to write version 2 of the meta.properties file? I am trying to > > understand when Kafka will write version 2 of the meta.properties > > file. I would assume that this is based on the MV but I don't see this > > stated explicitly in the KIP. When the MV changes to a version that > > supports meta.properties version 2 will Kafka automatically upgrade > > the version even if there are no changes to the state stored in > > meta.properties? > > > > JS3: > > > > How long will Kafka wait for the cluster id to be discovered? What > > happens if the cluster id is not in the cluster metadata partition and > > all of the meta.properties files? > > > > Thanks, > > -- > > -José > > -- -José
