Hi José, I think you make a valid point that our guarantees here are not actually different from zookeeper. In both systems, if you lose quorum, you will probably lose some data. Of course, how much data you lose depends on luck. If the last node standing was the active controller / zookeeper, then you got lucky.
This is why a lot of people run with 5 or 7 node quorums in both systems. The redundnancy is useful. I do think it would be nice to document some way of "forcing" the quorum into a specific configuration for data loss scenarios like this. This could also be used in the case where we lost 100% of the controllers. The brokers have a metadata snapshot and metadata log, so in an emergency you could grab the metadata from there. Perhaps something like "format with existing metadata"? If we did something like that, we should probably make it a separate tool from the formatting tool, and explicitly make it interactive (requires you to type "YES" on the console or something), since I DON'T want the folks making docker images and so on to do this. best, Colin On Mon, Apr 7, 2025, at 14:26, José Armando García Sancio wrote: > Thanks Luke. > > On Thu, Apr 3, 2025 at 7:14 AM Luke Chen <show...@gmail.com> wrote: >> In addition to the approaches you provided, maybe we can have a way to >> "force" KRaft to honor "controller.quorum.voters" config, instead of >> "controller.quorum.bootstrap.servers", even it's in kraft.version 1. > > Small clarification. In KIP-595, controller.quorum.voters was playing > two roles. 1) the set voters used by KRaft during HWM calculation and > leader election. 2) the set of endpoints used by observers (brokers) > to discover the leader (active controller). > > In KIP-853, I split that functionality. The set of voters was moved to > control records (VotersRecord) in the cluster metadata partition. The > bootstrap endpoints used by observer/brokers to discover the leader > were moved to the controller.quorum.bootstrap.servers property. > > I say that because Kafka supports using both configurations at the > same time. This is useful when upgrading from kraft.version 0 to > kraft.version 1. The upgrade process is roughly as follow: > 1. Add the controller.quorum.bootstrap.servers to all of the nodes > (controllers and brokers). > 2. Upgrade the kraft.version from 0 to 1. > 3. Monitor the "ignored-static-voters" metrics and remove the > controller.quorum.voters when the metric is 1. > > Thanks, > -- > -José
