On Fri, Apr 11, 2025, at 08:35, Andrew Schofield wrote: > Hi Colin, > Thanks for the KIP. >
Hi Andrew. Thanks for looking at the KIP. > AS1: I think that the KIP is not going to prove that useful because of > the introduction of new features as > the recent releases have occurred. I'll give a couple of examples. > > ELR was introduced as a preview in AK 4.0 and a feature enabled by > default in AK 4.1. The AK 4.0 binaries > understand how to do ELR, but earlier versions would not. Doesn't that > mean that it would be > disallowed to downgrade the MV beyond 4.0 if ELR had been enabled on > the cluster? No. Going from a metadata.version that supports ELR to one which does not would just require FeatureUpdate.UpgradeType.UNSAFE_DOWNGRADE. UNSAFE_DOWNGRADE is required whenever a downgrade would lose metadata, as discussed in the KIP. > We have a similar situation occurring with KIP-932 in the future > (preview in AK 4.1 using a feature, enabled by default in AK 4.2). I don't see any mention of metadata.version in KIP-932. As far as I can tell from skimming that KIP, users can downgrade metadata.version without affecting KIP-932. > And again with KIP-1071 (preview in AK 4.2 using a feature, > enabled by default in AK 4.3, I believe is the plan). I again don't see any relationship between KIP-1071 and metadata.version. KIP-1071 does not seem to care about metadata.version. > Similarly, would we need to be able to re-enable the old group coordinator > even though it doesn't > understand the new records on __consumer_offsets? The group.version feature, as described in KIP-848, doesn't depend on any particular metadata.verison. > So, my point is that the scope of downgrades which are safe or even > possible is very limited and the prerequisites of features which we > already have defined mean that we might need to disable features > which were previously defaulted on, and potentially clean up behind > them so that they properly re-initialize if the cluster is subsequently > upgraded back to a point where the features are re-enabled. To be clear, this KIP is not a generic downgrade mechanism for any KIP-584 feature. It's very specifically for metadata.version. If you ask to downgrade metadata.version, and you can downgrade metadata.version, we will do it for you. That's it. It doesn't mean group.version will be downgraded, or any other KIP-584 feature. Just metadata.version. > I worry that we are making a combinatorial testing situation which will > be expensive. There is already a combinatorial testing situation, because different KIP-584 features can be set independently from each other. This is simply the tradeoff of having multiple features rather than a single big one. That is one reason why if someone is implementing a small feature, I usually tell them just to tie it to a specific metadata version. However, for things like the group coordinator, it wouldn't really make sense to tie to a metadata.version, since it's so separate. Hope this helps. best, Colin > > Thanks, > Andrew > > ________________________________________ > From: Colin McCabe <cmcc...@apache.org> > Sent: 08 April 2025 20:46 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: Re: KIP-1155: Metadata Version Downgrades [DISCUSS] > > On Tue, Apr 8, 2025, at 09:55, Justine Olshan wrote: >> Hey PoAn, >> >> Just wanted to make a clarification. TV2 doesn't need to be downgraded if >> the MV is lower than the bootstrap MV for the TV. >> The bootstrap MV is only used when the cluster is created to get the >> default feature versions. As long as the TV version is a production version >> for the Kafka release, you can change MV to whatever you choose. It is >> decoupled from the MV at this point. >> >> Thanks, >> Justine >> > > Hi Justine, > > Thanks for the clarification. Yes, the "default versions" of features > don't necessarily imply hard dependencies. But if there are hard > dependencies (where a feature needs a certain MV), we will enforce that. > > best, > Colin > > >> On Tue, Apr 8, 2025 at 2:01 AM PoAn Yang <yangp...@gmail.com> wrote: >> >>> Hi Colin, >>> >>> Thanks for the KIP. It’s good to give users more flexibility to downgrade >>> metadata.version. >>> >>> PY1: Each feature has bootstrapMetadataVersion. For example, TV_2 is >>> IBP_4_0_IV2. >>> Assume that a cluster uses TV_2 and metadata.version 4.0-IV2. If a user >>> downgrades >>> metadata.version to 3.9-IV0, will the controller check TransactionVersion >>> and reject >>> the request? >>> >>> PY2: After users downgrade metadata.version, can they also restart server >>> with old binary >>> version? If they can, we need to add related end-to-end ducktape test. >>> >>> Thanks, >>> PoAn >>> >>> > On Apr 8, 2025, at 6:04 AM, Colin McCabe <cmcc...@apache.org> wrote: >>> > >>> > Hi all, >>> > >>> > I posted a new KIP about support metadata.version downgrades. Take a >>> look here: >>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1155%3A+Metadata+Version+Downgrades >>> > >>> > thanks, >>> > Colin >>> >>>
