If we implement the topic-level validation first, we will run into the same backward compatibility issues. I will start a KIP shortly to address the validation constraints holistically
> Jun Rao via dev <[email protected]> 於 2026年2月28日 凌晨2:27 寫道： > > Hi, Chia-Ping, > > Thanks for the proposal. It could work. Do you want to flesh that out as a > separate KIP? > > What's your suggestion for https://issues.apache.org/jira/browse/KAFKA-20107? > Should we implement it before or after your proposal is in place? > > Thanks, > > Jun > > > >> On Thu, Feb 19, 2026 at 12:03 AM Chia-Ping Tsai <[email protected]> wrote: >> >> A rough idea is that we could add a tagged field to the broker >> registration to contain the static properties that have validators in their >> config definition. The goal is to empower the controller to do more >> validation before approving the upgrade >> >>> On 2026/02/17 19:28:08 Jun Rao via dev wrote: >>> Hi, Chia-Ping, >>> >>> Thanks for the reply. >>> >>> I am not sure about the MV approach. For example, a user could set >>> `segment.bytes` through a static config file. How will MV work in that >> case? >>> >>> Jun >>> >>> On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <[email protected]> >> wrote: >>> >>>> Hi, Jun >>>> >>>> I prefer to ensure users are aware of what happens with their >>>> configuration. Clamping seems to hide the actual behaviour from users, >>>> which may lead to potential issues. >>>> >>>> Here is a crazy idea: what if the validation is bound to the MV? >>>> >>>> For example, `segment.bytes` could have different lower bounds for 4.0 >> and >>>> 4.1. This allows users to safely keep the smaller value while >> upgrading the >>>> binary files. However, they must update the configurations to meet the >> new >>>> requirement before bumping the MV. Otherwise, they will receive an >> error >>>> when calling `updateFeatures` >>>> >>>> This approach not only offers a smooth upgrade path bug also ensures >> the >>>> final cluster state adheres to the strict validation rules once the MV >> is >>>> updated. >>>> >>>> WDYT? >>>> >>>> Best, >>>> Chia-Ping >>>> >>>> On 2026/02/09 17:08:19 Jun Rao via dev wrote: >>>>> Hi, Chia-Ping, >>>>> >>>>> Thanks for the reply. Do you agree that it's better to use the >> clamping >>>>> approach for segment.bytes ? >>>>> >>>>> Jun >>>>> >>>>> On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <[email protected]> >>>> wrote: >>>>> >>>>>> hi Jun >>>>>> >>>>>> I agree that we should apply reasonable strategies for different >>>> `breaking >>>>>> changes`. This makes ensuring that the initialization process sees >> the >>>>>> 'latest' configs even more critical. It is otherwise hard to >> guarantee >>>> that >>>>>> the broker fails at the appropriate time when a `fail-fast` policy >> is >>>>>> intended >>>>>> >>>>>> Best, >>>>>> Chia-Ping >>>>>> >>>>>> On 2026/01/30 22:13:00 Jun Rao via dev wrote: >>>>>>> Hi, Chia-Ping, >>>>>>> >>>>>>> Thanks for following up on this. >>>>>>> >>>>>>> To me, there are two categories of changes. The first type >> involves >>>>>> changes >>>>>>> that always benefit the user. An example is segment.bytes since >>>> there is >>>>>> no >>>>>>> good reason for a user to ever set it to less than 1MB. In this >>>> case, one >>>>>>> option is to automatically clamp the value to the new lower bound >>>> without >>>>>>> failing. This is probably best for users because they get better >>>>>>> configuration and compatibility. This is essentially the approach >>>> that we >>>>>>> followed in KIP-1161 for handling duplicates in the List config. >> The >>>>>> second >>>>>>> category includes changes where disallowed values may impact a >> user's >>>>>>> intention. An example is segment.ms (change not implemented >> yet, but >>>>>>> targeted for 5.0). A user may intentionally set it to a really >> small >>>>>> value >>>>>>> to control the retention time. In this case, it's probably >> better to >>>> fail >>>>>>> the broker with an invalid config error so that the users know >> about >>>> it. >>>>>>> >>>>>>> Jun >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai < >> [email protected] >>>>> >>>>>> wrote: >>>>>>> >>>>>>>> hi all >>>>>>>> >>>>>>>> There was a long discussion about compatibility after >> increasing >>>> the >>>>>> lower >>>>>>>> bound ( >>>>>> https://github.com/apache/kafka/pull/20334#discussion_r2538011530 >> ). >>>>>>>> >>>>>>>> In summary, increasing the lower bound results in the following >>>> issues: >>>>>>>> >>>>>>>> 1) Static configurations created before KIP-1030 cause the >> broker >>>> to >>>>>> fail. >>>>>>>> For example, setting log.segment.bytes=1000 will now encounter >> a >>>>>> validation >>>>>>>> error due to the new lower bound when starting the updated >> broker. >>>>>>>> >>>>>>>> 2) Dynamic configurations created before KIP-1030 cannot be >> applied >>>>>> due to >>>>>>>> validation errors. >>>>>>>> >>>>>>>> Personally, I believe users should be explicitly aware of >> breaking >>>>>>>> changes. Therefore, I suggest forcing the broker to fail when >>>>>> initializing >>>>>>>> the metadata publisher if the dynamic configuration cannot be >>>> applied. >>>>>>>> >>>>>>>> A softer approach is to use warnings instead of fatal errors. >> The >>>>>> broker >>>>>>>> would proceed smoothly, but users might be unaware of the >> breaking >>>>>> changes, >>>>>>>> and the broker would run with unexpected configurations. >>>>>>>> >>>>>>>> Any feedback is welcome. >>>>>>>> >>>>>>>> Best, >>>>>>>> Chia-Ping >>>>>>>> >>>>>>>> On 2024/11/18 10:13:41 Divij Vaidya wrote: >>>>>>>>> Hey folks >>>>>>>>> >>>>>>>>> With 4.0, we have an opportunity to reset the default values >> and >>>> add >>>>>>>>> constraints in the configurations based on our learnings >> since >>>> 3.0. >>>>>>>>> >>>>>>>>> Here's a KIP which modifies defaults for some properties and >>>>>> modifies the >>>>>>>>> constraints for a few others. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations >>>>>>>>> >>>>>>>>> >>>>>>>>> Looking forward for your feedback. >>>>>>>>> >>>>>>>>> (Previous discussion thread on this topic - >>>>>>>>> >> https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp >>>> ) >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
