A rough idea is that we could add a tagged field to the broker registration to contain the static properties that have validators in their config definition. The goal is to empower the controller to do more validation before approving the upgrade
On 2026/02/17 19:28:08 Jun Rao via dev wrote: > Hi, Chia-Ping, > > Thanks for the reply. > > I am not sure about the MV approach. For example, a user could set > `segment.bytes` through a static config file. How will MV work in that case? > > Jun > > On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <[email protected]> wrote: > > > Hi, Jun > > > > I prefer to ensure users are aware of what happens with their > > configuration. Clamping seems to hide the actual behaviour from users, > > which may lead to potential issues. > > > > Here is a crazy idea: what if the validation is bound to the MV? > > > > For example, `segment.bytes` could have different lower bounds for 4.0 and > > 4.1. This allows users to safely keep the smaller value while upgrading the > > binary files. However, they must update the configurations to meet the new > > requirement before bumping the MV. Otherwise, they will receive an error > > when calling `updateFeatures` > > > > This approach not only offers a smooth upgrade path bug also ensures the > > final cluster state adheres to the strict validation rules once the MV is > > updated. > > > > WDYT? > > > > Best, > > Chia-Ping > > > > On 2026/02/09 17:08:19 Jun Rao via dev wrote: > > > Hi, Chia-Ping, > > > > > > Thanks for the reply. Do you agree that it's better to use the clamping > > > approach for segment.bytes ? > > > > > > Jun > > > > > > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <[email protected]> > > wrote: > > > > > > > hi Jun > > > > > > > > I agree that we should apply reasonable strategies for different > > `breaking > > > > changes`. This makes ensuring that the initialization process sees the > > > > 'latest' configs even more critical. It is otherwise hard to guarantee > > that > > > > the broker fails at the appropriate time when a `fail-fast` policy is > > > > intended > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > On 2026/01/30 22:13:00 Jun Rao via dev wrote: > > > > > Hi, Chia-Ping, > > > > > > > > > > Thanks for following up on this. > > > > > > > > > > To me, there are two categories of changes. The first type involves > > > > changes > > > > > that always benefit the user. An example is segment.bytes since > > there is > > > > no > > > > > good reason for a user to ever set it to less than 1MB. In this > > case, one > > > > > option is to automatically clamp the value to the new lower bound > > without > > > > > failing. This is probably best for users because they get better > > > > > configuration and compatibility. This is essentially the approach > > that we > > > > > followed in KIP-1161 for handling duplicates in the List config. The > > > > second > > > > > category includes changes where disallowed values may impact a user's > > > > > intention. An example is segment.ms (change not implemented yet, but > > > > > targeted for 5.0). A user may intentionally set it to a really small > > > > value > > > > > to control the retention time. In this case, it's probably better to > > fail > > > > > the broker with an invalid config error so that the users know about > > it. > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <[email protected] > > > > > > > wrote: > > > > > > > > > > > hi all > > > > > > > > > > > > There was a long discussion about compatibility after increasing > > the > > > > lower > > > > > > bound ( > > > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530). > > > > > > > > > > > > In summary, increasing the lower bound results in the following > > issues: > > > > > > > > > > > > 1) Static configurations created before KIP-1030 cause the broker > > to > > > > fail. > > > > > > For example, setting log.segment.bytes=1000 will now encounter a > > > > validation > > > > > > error due to the new lower bound when starting the updated broker. > > > > > > > > > > > > 2) Dynamic configurations created before KIP-1030 cannot be applied > > > > due to > > > > > > validation errors. > > > > > > > > > > > > Personally, I believe users should be explicitly aware of breaking > > > > > > changes. Therefore, I suggest forcing the broker to fail when > > > > initializing > > > > > > the metadata publisher if the dynamic configuration cannot be > > applied. > > > > > > > > > > > > A softer approach is to use warnings instead of fatal errors. The > > > > broker > > > > > > would proceed smoothly, but users might be unaware of the breaking > > > > changes, > > > > > > and the broker would run with unexpected configurations. > > > > > > > > > > > > Any feedback is welcome. > > > > > > > > > > > > Best, > > > > > > Chia-Ping > > > > > > > > > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote: > > > > > > > Hey folks > > > > > > > > > > > > > > With 4.0, we have an opportunity to reset the default values and > > add > > > > > > > constraints in the configurations based on our learnings since > > 3.0. > > > > > > > > > > > > > > Here's a KIP which modifies defaults for some properties and > > > > modifies the > > > > > > > constraints for a few others. > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations > > > > > > > > > > > > > > > > > > > > > Looking forward for your feedback. > > > > > > > > > > > > > > (Previous discussion thread on this topic - > > > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp > > ) > > > > > > > > > > > > > > > > > > > > > > > > > > > >
