Hey Satish, someone reported a minor bug in the Streams application shutdown which was a recent regression, though not strictly a new one: was introduced in 3.4 I believe.
The fix seems to be super lightweight and low-risk so I was hoping to slip it into 3.6 if that's ok with you? They plan to have the patch tonight. https://issues.apache.org/jira/browse/KAFKA-15429 On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana <satish.dugg...@gmail.com> wrote: > Thanks Chris for bringing this issue here and filing the new JIRA for > 3.6.0[1]. It seems to be a blocker for 3.6.0. > > Please help review https://github.com/apache/kafka/pull/14314 as Chris > requested. > > 1. https://issues.apache.org/jira/browse/KAFKA-15425 > > ~Satish. > > On Fri, 1 Sept 2023 at 03:59, Chris Egerton <chr...@aiven.io.invalid> > wrote: > > > > Hi all, > > > > Quick update: I've filed a separate ticket, > > https://issues.apache.org/jira/browse/KAFKA-15425, to track the behavior > > change in Admin::listOffsets. For the full history of the ticket, it's > > worth reading the comment thread on the old ticket at > > https://issues.apache.org/jira/browse/KAFKA-12879. > > > > I've also published https://github.com/apache/kafka/pull/14314 as a > fairly > > lightweight PR to revert the behavior of Admin::listOffsets without also > > reverting the refactoring to use the internal admin driver API. Would > > appreciate a review on that if anyone can spare the cycles. > > > > Cheers, > > > > Chris > > > > On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton <chr...@aiven.io> wrote: > > > > > Hi Satish, > > > > > > Wanted to let you know that KAFKA-12879 ( > > > https://issues.apache.org/jira/browse/KAFKA-12879), a breaking change > in > > > Admin::listOffsets, has been reintroduced into the code base. Since we > > > haven't yet published a release with this change (at least, not the > more > > > recent instance of it), I was hoping we could treat it as a blocker for > > > 3.6.0. I'd also like to solicit the input of people familiar with the > admin > > > client to weigh in on the Jira ticket about whether we should continue > to > > > preserve the current behavior (if the consensus is that we should, I'm > > > happy to file a fix). > > > > > > Please let me know if you agree that this qualifies as a blocker. I > plan > > > on publishing a potential fix sometime this week. > > > > > > Cheers, > > > > > > Chris > > > > > > On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana < > satish.dugg...@gmail.com> > > > wrote: > > > > > >> Hi, > > >> Please plan to continue merging pull requests associated with any > > >> outstanding minor features and stabilization changes to 3.6 branch > > >> before September 3rd. Kindly update the KIP's implementation status in > > >> the 3.6.0 release notes. > > >> > > >> Thanks, > > >> Satish. > > >> > > >> On Fri, 25 Aug 2023 at 21:37, Justine Olshan > > >> <jols...@confluent.io.invalid> wrote: > > >> > > > >> > Hey Satish, > > >> > Everything should be in 3.6, and I will update the release plan > wiki. > > >> > Thanks! > > >> > > > >> > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana < > > >> satish.dugg...@gmail.com> > > >> > wrote: > > >> > > > >> > > Hi Justine, > > >> > > Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part > looks > > >> > > to be addressing a critical issue of consumers getting stuck. > Please > > >> > > update the release plan wiki and merge all the required changes > to 3.6 > > >> > > branch. > > >> > > > > >> > > Thanks, > > >> > > Satish. > > >> > > > > >> > > On Thu, 24 Aug 2023 at 22:19, Justine Olshan > > >> > > <jols...@confluent.io.invalid> wrote: > > >> > > > > > >> > > > Hey Satish, > > >> > > > Does it make sense to include KIP-890 part 1? It prevents > hanging > > >> > > > transactions for older clients. (An optimization and stronger > EOS > > >> > > > guarantees will be included in part 2) > > >> > > > > > >> > > > Thanks, > > >> > > > Justine > > >> > > > > > >> > > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana < > > >> satish.dugg...@gmail.com > > >> > > > > > >> > > > wrote: > > >> > > > > > >> > > > > Hi, > > >> > > > > 3.6 branch is created. Please make sure any PRs targeted for > 3.6.0 > > >> > > > > should be merged to 3.6 branch once those are merged to trunk. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > Satish. > > >> > > > > > > >> > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana < > > >> satish.dugg...@gmail.com > > >> > > > > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > Hi, > > >> > > > > > Please plan to merge PRs(including the major features) > targeted > > >> for > > >> > > > > > 3.6.0 by the end of Aug 20th UTC. Starting from August 21st, > > >> any pull > > >> > > > > > requests intended for the 3.6.0 release must include the > changes > > >> > > > > > merged into the 3.6 branch as mentioned in the release plan. > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Satish. > > >> > > > > > > > >> > > > > > On Fri, 4 Aug 2023 at 18:39, Chris Egerton > > >> <chr...@aiven.io.invalid> > > >> > > > > wrote: > > >> > > > > > > > > >> > > > > > > Thanks for adding KIP-949, Satish! > > >> > > > > > > > > >> > > > > > > On Fri, Aug 4, 2023 at 7:06 AM Satish Duggana < > > >> > > > > satish.dugg...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Hi, > > >> > > > > > > > Myself and Divij discussed and added the wiki for Kafka > > >> > > TieredStorage > > >> > > > > > > > Early Access Release[1]. If you have any comments or > > >> feedback, > > >> > > please > > >> > > > > > > > feel free to share them. > > >> > > > > > > > > > >> > > > > > > > 1. > > >> > > > > > > > > > >> > > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes > > >> > > > > > > > > > >> > > > > > > > Thanks, > > >> > > > > > > > Satish. > > >> > > > > > > > > > >> > > > > > > > On Fri, 4 Aug 2023 at 08:40, Satish Duggana < > > >> > > > > satish.dugg...@gmail.com> > > >> > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > Hi Chris, > > >> > > > > > > > > Thanks for the update. This looks to be a minor change > > >> and is > > >> > > also > > >> > > > > > > > > useful for backward compatibility. I added it to the > > >> release > > >> > > plan > > >> > > > > as > > >> > > > > > > > > an exceptional case. > > >> > > > > > > > > > > >> > > > > > > > > ~Satish. > > >> > > > > > > > > > > >> > > > > > > > > On Thu, 3 Aug 2023 at 21:34, Chris Egerton > > >> > > <chr...@aiven.io.invalid > > >> > > > > > > > >> > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > Hi Satish, > > >> > > > > > > > > > > > >> > > > > > > > > > Would it be possible to include KIP-949 ( > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy > > >> > > > > > > > ) > > >> > > > > > > > > > in the 3.6.0 release? It passed voting yesterday, > and > > >> is a > > >> > > very > > >> > > > > small, > > >> > > > > > > > > > low-risk change that we'd like to put out as soon as > > >> > > possible in > > >> > > > > order > > >> > > > > > > > to > > >> > > > > > > > > > patch an accidental break in backwards compatibility > > >> caused > > >> > > a few > > >> > > > > > > > versions > > >> > > > > > > > > > ago. > > >> > > > > > > > > > > > >> > > > > > > > > > Best, > > >> > > > > > > > > > > > >> > > > > > > > > > Chris > > >> > > > > > > > > > > > >> > > > > > > > > > On Fri, Jul 28, 2023 at 2:35 AM Satish Duggana < > > >> > > > > > > > satish.dugg...@gmail.com> > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Hi All, > > >> > > > > > > > > > > Whoever has KIP entries in the 3.6.0 release plan. > > >> Please > > >> > > > > update it > > >> > > > > > > > > > > with the latest status by tomorrow(end of the day > > >> 29th Jul > > >> > > UTC > > >> > > > > ). > > >> > > > > > > > > > > > > >> > > > > > > > > > > Thanks > > >> > > > > > > > > > > Satish. > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Fri, 28 Jul 2023 at 12:01, Satish Duggana < > > >> > > > > > > > satish.dugg...@gmail.com> > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks Ismael and Divij for the suggestions. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > One way was to follow the earlier guidelines > that > > >> we set > > >> > > for > > >> > > > > any > > >> > > > > > > > early > > >> > > > > > > > > > > > access release. It looks Ismael already > mentioned > > >> the > > >> > > > > example of > > >> > > > > > > > > > > > KRaft. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > KIP-405 mentions upgrade/downgrade and > limitations > > >> > > sections. > > >> > > > > We can > > >> > > > > > > > > > > > clarify that in the release notes for users on > how > > >> this > > >> > > > > feature > > >> > > > > > > > can be > > >> > > > > > > > > > > > used for early access. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Divij, We do not want users to enable this > feature > > >> on > > >> > > > > production > > >> > > > > > > > > > > > environments in early access release. Let us > work > > >> > > together > > >> > > > > on the > > >> > > > > > > > > > > > followups Ismael suggested. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > ~Satish. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Fri, 28 Jul 2023 at 02:24, Divij Vaidya < > > >> > > > > > > > divijvaidy...@gmail.com> > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Those are great suggestions, thank you. We > will > > >> > > continue > > >> > > > > this > > >> > > > > > > > > > > discussion > > >> > > > > > > > > > > > > forward in a separate KIP for release plan for > > >> Tiered > > >> > > > > Storage. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Thu 27. Jul 2023 at 21:46, Ismael Juma < > > >> > > > > m...@ismaeljuma.com> > > >> > > > > > > > wrote: > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Hi Divij, > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I think the points you bring up for > discussion > > >> are > > >> > > all > > >> > > > > good. > > >> > > > > > > > My main > > >> > > > > > > > > > > > > > feedback is that they should be discussed > in the > > >> > > context > > >> > > > > of > > >> > > > > > > > KIPs vs > > >> > > > > > > > > > > the > > >> > > > > > > > > > > > > > release template. That's why we have a > backwards > > >> > > > > compatibility > > >> > > > > > > > > > > section for > > >> > > > > > > > > > > > > > every KIP, it's precisely to ensure we think > > >> > > carefully > > >> > > > > about > > >> > > > > > > > some of > > >> > > > > > > > > > > the > > >> > > > > > > > > > > > > > points you're bringing up. When it comes to > > >> defining > > >> > > the > > >> > > > > > > > meaning of > > >> > > > > > > > > > > early > > >> > > > > > > > > > > > > > access, we have two options: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > 1. Have a KIP specifically for tiered > storage. > > >> > > > > > > > > > > > > > 2. Have a KIP to define general guidelines > for > > >> what > > >> > > early > > >> > > > > > > > access > > >> > > > > > > > > > > means. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Does this make sense? > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Ismael > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 6:38 PM Divij > Vaidya < > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thank you for the response, Ismael. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 1. Specifically in context of 3.6, I > wanted > > >> this > > >> > > > > > > > compatibility > > >> > > > > > > > > > > > > > > guarantee point to encourage a discussion > on > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > >> > > > > > > > > > > > > > > . > > >> > > > > > > > > > > > > > > Due to lack of producer snapshots in <2.8 > > >> > > versions, a > > >> > > > > > > > customer may > > >> > > > > > > > > > > not > > >> > > > > > > > > > > > > > > be able to upgrade to 3.6 and use TS on a > > >> topic > > >> > > which > > >> > > > > was > > >> > > > > > > > created > > >> > > > > > > > > > > when > > >> > > > > > > > > > > > > > > the cluster was on <2.8 version (see > > >> motivation for > > >> > > > > > > > details). We > > >> > > > > > > > > > > can > > >> > > > > > > > > > > > > > > discuss and agree that it does not break > > >> > > compatibility, > > >> > > > > > > > which is > > >> > > > > > > > > > > fine. > > >> > > > > > > > > > > > > > > But I want to ensure that we have a > discussion > > >> > > soon on > > >> > > > > this > > >> > > > > > > > to > > >> > > > > > > > > > > reach a > > >> > > > > > > > > > > > > > > conclusion. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 2. I will start a KIP on this for further > > >> > > discussion. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 3. In the context of 3.6, this would mean > that > > >> > > there > > >> > > > > should > > >> > > > > > > > be > > >> > > > > > > > > > > > > > > no-regression, if a user does "not" > turn-on > > >> remote > > >> > > > > storage > > >> > > > > > > > (early > > >> > > > > > > > > > > > > > > access feature) at a cluster level. We > have > > >> some > > >> > > known > > >> > > > > cases > > >> > > > > > > > (such > > >> > > > > > > > > > > as > > >> > > > > > > > > > > > > > > > > >> https://issues.apache.org/jira/browse/KAFKA-15189) > > >> > > > > which > > >> > > > > > > > violate > > >> > > > > > > > > > > this > > >> > > > > > > > > > > > > > > compatibility requirement. Having this > > >> guarantee > > >> > > > > mentioned > > >> > > > > > > > in the > > >> > > > > > > > > > > > > > > release plan will ensure that we are all > in > > >> > > agreement > > >> > > > > with > > >> > > > > > > > which > > >> > > > > > > > > > > cases > > >> > > > > > > > > > > > > > > are truly blockers and which aren't. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 4. Fair, instead of a general goal, let me > > >> put it > > >> > > > > > > > specifically in > > >> > > > > > > > > > > the > > >> > > > > > > > > > > > > > > context of 3.6. Let me know if this is > not the > > >> > > right > > >> > > > > forum > > >> > > > > > > > for this > > >> > > > > > > > > > > > > > > discussion. > > >> > > > > > > > > > > > > > > Once a user "turns on" tiered storage > (TS) at > > >> a > > >> > > cluster > > >> > > > > > > > level, I am > > >> > > > > > > > > > > > > > > proposing that they should have the > ability to > > >> > > turn it > > >> > > > > off > > >> > > > > > > > as well > > >> > > > > > > > > > > at > > >> > > > > > > > > > > > > > > a cluster level. Since this is a topic > level > > >> > > feature, > > >> > > > > folks > > >> > > > > > > > may not > > >> > > > > > > > > > > > > > > spin up a separate cluster to try this > > >> feature, > > >> > > hence, > > >> > > > > we > > >> > > > > > > > need to > > >> > > > > > > > > > > > > > > ensure that we provide them with the > ability > > >> to try > > >> > > > > tiered > > >> > > > > > > > storage > > >> > > > > > > > > > > for > > >> > > > > > > > > > > > > > > a topic which could be deleted and > featured > > >> > > > > turned-off, so > > >> > > > > > > > that > > >> > > > > > > > > > > rest > > >> > > > > > > > > > > > > > > of the production cases are not impacted. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 5. Agree on not making public interface > > >> change as a > > >> > > > > > > > requirement > > >> > > > > > > > > > > but we > > >> > > > > > > > > > > > > > > should define what "early access" means in > > >> that > > >> > > case. > > >> > > > > Users > > >> > > > > > > > may > > >> > > > > > > > > > > not be > > >> > > > > > > > > > > > > > > aware that "early access" public APIs may > > >> change > > >> > > > > (unless I am > > >> > > > > > > > > > > missing > > >> > > > > > > > > > > > > > > some documentation somewhere completely, > in > > >> which > > >> > > case > > >> > > > > I > > >> > > > > > > > apologize > > >> > > > > > > > > > > for > > >> > > > > > > > > > > > > > > bringing this naive point). > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > -- > > >> > > > > > > > > > > > > > > Divij Vaidya > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael > Juma < > > >> > > > > > > > m...@ismaeljuma.com> > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Hi Divij, > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Some of these are launch checklist items > > >> (not > > >> > > really > > >> > > > > > > > goals) and > > >> > > > > > > > > > > some > > >> > > > > > > > > > > > > > are > > >> > > > > > > > > > > > > > > > compatibility guarantees. More below. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij > Vaidya > > >> < > > >> > > > > > > > > > > divijvaidy...@gmail.com> > > >> > > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Hey Satish > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Could we consider adding "launch > goals" > > >> in the > > >> > > > > release > > >> > > > > > > > plan. > > >> > > > > > > > > > > While > > >> > > > > > > > > > > > > > > > > some of these may be implicit, it > would be > > >> > > nice to > > >> > > > > list > > >> > > > > > > > them > > >> > > > > > > > > > > down in > > >> > > > > > > > > > > > > > > > > the release plan. For this release, > our > > >> launch > > >> > > > > > > > requirements > > >> > > > > > > > > > > would be: > > >> > > > > > > > > > > > > > > > > 1. Users should be able to upgrade > from > > >> any > > >> > > prior > > >> > > > > Kafka > > >> > > > > > > > > > > version to > > >> > > > > > > > > > > > > > this > > >> > > > > > > > > > > > > > > > > version. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > This is part of the compatibility > > >> guarantees. The > > >> > > > > upgrade > > >> > > > > > > > notes > > >> > > > > > > > > > > mention > > >> > > > > > > > > > > > > > > > this already. If there is a change in a > > >> given > > >> > > > > release, it > > >> > > > > > > > should > > >> > > > > > > > > > > > > > > definitely > > >> > > > > > > > > > > > > > > > be highlighted. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > 2. On release, this version (or it's > > >> > > dependencies) > > >> > > > > would > > >> > > > > > > > not > > >> > > > > > > > > > > have any > > >> > > > > > > > > > > > > > > > > known MEDIUM/HIGH CVE. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > This is a new policy and the details > should > > >> be > > >> > > > > discussed. > > >> > > > > > > > In > > >> > > > > > > > > > > > > > particular, > > >> > > > > > > > > > > > > > > > the threshold (medium or high). > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > 3. Presence of any "early access"/"beta" > > >> feature > > >> > > > > should not > > >> > > > > > > > > > > impact > > >> > > > > > > > > > > > > > > > > other production features when it is > not > > >> > > enabled. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > This is a general guideline for early > access > > >> > > > > features and > > >> > > > > > > > not > > >> > > > > > > > > > > specific > > >> > > > > > > > > > > > > > to > > >> > > > > > > > > > > > > > > > this release. It would be good to have a > > >> page > > >> > > that > > >> > > > > talks > > >> > > > > > > > about > > >> > > > > > > > > > > these > > >> > > > > > > > > > > > > > > things. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > 4. Once enabled, users should have an > > >> option to > > >> > > > > disable any > > >> > > > > > > > > > > "early > > >> > > > > > > > > > > > > > > > > access"/"beta" feature and resume > normal > > >> > > production > > >> > > > > > > > features, > > >> > > > > > > > > > > i.e. > > >> > > > > > > > > > > > > > > > > impact of beta features should be > > >> reversible. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > This needs discussion and I don't think > it's > > >> > > > > reasonable as > > >> > > > > > > > a > > >> > > > > > > > > > > general > > >> > > > > > > > > > > > > > > rule. > > >> > > > > > > > > > > > > > > > For example, Kraft early access wasn't > > >> reversible > > >> > > > > and it > > >> > > > > > > > was not > > >> > > > > > > > > > > > > > feasible > > >> > > > > > > > > > > > > > > > for it to be. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > 5. KIP-405 will be available in "early > > >> > > access"/"beta" > > >> > > > > > > > mode. Early > > >> > > > > > > > > > > > > > > > > access/beta means that the public > facing > > >> > > > > interfaces won't > > >> > > > > > > > > > > change in > > >> > > > > > > > > > > > > > > > > future but the implementation is not > > >> > > recommended > > >> > > > > to be > > >> > > > > > > > used in > > >> > > > > > > > > > > > > > > > > production. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I don't think it's ok to make this a > > >> requirement. > > >> > > > > Early > > >> > > > > > > > access > > >> > > > > > > > > > > is a way > > >> > > > > > > > > > > > > > > to > > >> > > > > > > > > > > > > > > > get early feedback and all types of > changes > > >> > > should > > >> > > > > be on > > >> > > > > > > > the > > >> > > > > > > > > > > table. > > >> > > > > > > > > > > > > > They > > >> > > > > > > > > > > > > > > > would be discussed via KIPs as usual. I > > >> believe > > >> > > > > there were > > >> > > > > > > > some > > >> > > > > > > > > > > > > > > > incompatible changes for Kraft during > the > > >> early > > >> > > > > access > > >> > > > > > > > period > > >> > > > > > > > > > > although > > >> > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > > team aimed to minimize work required > during > > >> > > > > upgrades. I > > >> > > > > > > > have > > >> > > > > > > > > > > mentioned > > >> > > > > > > > > > > > > > > > Kraft a couple of times since it's a > good > > >> > > example of > > >> > > > > a > > >> > > > > > > > large > > >> > > > > > > > > > > feature > > >> > > > > > > > > > > > > > that > > >> > > > > > > > > > > > > > > > went through this process. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Ismael > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > -- > > >> > > > > > > > > > > > > Divij Vaidya > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > >> > > > >
