On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > I think using feature flags (whether we support a framework and tooling for > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to > this KIP. I think the value of this KIP is that it's trying to propose a > systemic approach for gating functionality that may take multiple releases > to develop. A problem with ad-hoc feature flags is that it's useful during > feature development, so that people who are working on this feature (or are > interested in beta-testing the feature) can get early access (without any > guarantees on compatibility or even correctness); but then the feature > flags often stick forever after the feature development is done (and as > time moves one, the new code is written in such a way that it's not > possible to turn the feature off any more cleanly). >
Hi Artem, I think feature flags are somewhat orthogonal to the stable / unstable discussion. Even if every new feature was a feature flag, you probably still wouldn't want to stabilize the features immediately, to avoid maintaining a lot of alpha stuff forever. (I also think that feature flags should be used sparingly, if at all, because of the way that they exponentially increase the test matrix. But that's a tangent, I think, given the first point...) > > I'd also clarify how I think about "stable". Ismael made a comment " > something is stable in the "this is battle-tested" sense.". I don't think > it has to be "battle-tested", it just has to meet the bar of being in the > trunk. Again, thinking of a small single-commit feature -- to commit to > trunk, the feature doesn't have to be "battle-tested", but it should be > complete (and not just a bunch of TODOs), with tests written and some level > of dev-testing done, so that once the release is cut, we can find and fix > bugs and make it release-quality (as opposed to reverting the whole > thing). We can apply the same "stability" bar for features to be in the > stable MV -- fully complete, tests written and some level of dev testing > done. > I'm struggling a bit with your phrasing. Are you suggesting that unstable MVs would not be able to be in trunk? I think we do want them to be able to go into trunk. If they have to go into a branch, then there is actually no need for any of this. Doing big features in long-lived branches is one genuine alternative to unstable MVs, I think. But it's not an alternative that works well with our current tooling for continuous integration, deployment, building, etc. I think it would also impact developer productivity somewhat negatively. best, Colin > > -Artem > > On Fri, Jan 12, 2024 at 10:12 AM Justine Olshan > <jols...@confluent.io.invalid> wrote: > >> Hi Ismael, >> >> I debated including something about feature flags in my last comment, but >> maybe I should have. >> What you said makes sense. >> >> Justine >> >> On Fri, Jan 12, 2024 at 9:31 AM Ismael Juma <m...@ismaeljuma.com> wrote: >> >> > Justine, >> > >> > For features that are not production-ready, they should have an >> additional >> > configuration (not the metadata version) that enables/disables it. The MV >> > specific features we ship are something we have to support and make sure >> we >> > don't break going forward. >> > >> > Ismael >> > >> > On Fri, Jan 12, 2024 at 9:26 AM Justine Olshan >> > <jols...@confluent.io.invalid> >> > wrote: >> > >> > > Hi Ismael, >> > > >> > > I think the concern I have about a MV for a feature that is not >> > production >> > > ready is that it blocks any development/features with higher MV >> versions >> > > that could be production ready. >> > > >> > > I do see your point though. Previously MV/IBP was about pure broker >> > > compatibility and not about the confidence in the feature it is >> gating. I >> > > do wonder though if it could be useful to have that sort of gating. >> > > I also wonder if an internal config could be useful so that the average >> > > user doesn't have to worry about the complexities of unstable metadata >> > > versions (and their risks). >> > > >> > > I am ok with options 2 and 2 as well by the way. >> > > >> > > Justine >> > > >> > > On Fri, Jan 12, 2024 at 7:36 AM Ismael Juma <m...@ismaeljuma.com> wrote: >> > > >> > > > Hi, >> > > > >> > > > Thanks for the KIP. >> > > > >> > > > Reading the discussion, I think a lot of the confusion is due to the >> > > > "unstable" naming. People are then trying to figure out when we think >> > > > something is stable in the "this is battle-tested" sense. But this >> flag >> > > > should not be about that. We can have an MV for a feature that is not >> > yet >> > > > production-ready (and we did that when KRaft itself was not >> production >> > > > ready). I think this flag is about metadata versions that are >> basically >> > > > unsupported - if you use it, you get to keep the pieces. They exist >> > > solely >> > > > to make the lives of Apache Kafka developers easier. I would even >> > suggest >> > > > that the config we introduce be of the internal variety, ie it won't >> > show >> > > > in the generated documentation and there won't be any compatibility >> > > > guarantee. >> > > > >> > > > Thoughts? >> > > > >> > > > Ismael >> > > > >> > > > On Fri, Jan 5, 2024 at 7:33 AM Proven Provenzano >> > > > <pprovenz...@confluent.io.invalid> wrote: >> > > > >> > > > > Hey folks, >> > > > > >> > > > > I am starting a discussion thread for managing unstable metadata >> > > versions >> > > > > in Apache Kafka. >> > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka >> > > > > >> > > > > This KIP is actually already implemented in 3.7 with PR >> > > > > https://github.com/apache/kafka/pull/14860. >> > > > > I have created this KIP to explain the motivation and how managing >> > > > Metadata >> > > > > Versions is expected to work. >> > > > > Comments are greatly appreciated as this process can always be >> > > improved. >> > > > > >> > > > > -- >> > > > > --Proven >> > > > > >> > > > >> > > >> > >>
