Re: Apache Kafka 3.7.0 Release

Stanislav this *doesn't* seem like a blocker to me given the complexity, > public API changes and Fair enough, since it only has been seen for very high partition counts. The PR in trunk is under active review, i will cherry-pick it for 3.7.1. >

Re: Apache Kafka 3.7.0 Release

Hey Divij, that is a good point regarding KIP-848. David, as the author of the KIP, would you be able to drive this? Similarly, would anybody be willing to drive such an EA Release Note for the JBOD feature? Mayank, this *doesn't* seem like a blocker to me given the complexity, public API

Re: Apache Kafka 3.7.0 Release

Hi Folks KIP-951 was delivered fully in AK 3.7. Its 1st optimisation was delivered in 3.6.1, to skip backoff period for a produce batch being retried to new leader i.e. KAFKA-15415. KAFKA-15415 current implementation introduced a performance regression, by increasing synchronization on the

Re: Apache Kafka 3.7.0 Release

Hey folks The release plan for 3.7.0 [1] calls out KIP 848 as "Targeting a Preview in 3.7". Is that still true? If yes, then we should perhaps add that in the blog, call it out in the release notes and prepare a preview document similar to what we did for Tiered Storage Early Access release[2]

Re: Apache Kafka 3.7.0 Release

Hi all, There is a bug KAFKA-16101 reporting that "Kafka cluster will be unavailable during KRaft migration rollback". The impact for this issue is that if brokers try to rollback to ZK mode during KRaft migration process, there will be a period

Re: Apache Kafka 3.7.0 Release

Thanks Colin, With that, I believe we are out of blockers. I was traveling today and couldn't build an RC - expect one to be published tomorrow (barring any problems). In the meanwhile - here is a PR for the 3.7 blog post - https://github.com/apache/kafka-site/pull/578 Best, Stan On Wed, Jan

Re: Apache Kafka 3.7.0 Release

KAFKA-16094 has been fixed and backported to 3.7. Colin On Mon, Jan 8, 2024, at 14:52, Colin McCabe wrote: > On an unrelated note, I found a blocker bug related to upgrades from > 3.6 (and earlier) to 3.7. > > The JIRA is here: > https://issues.apache.org/jira/browse/KAFKA-16094 > > Fix

Re: Apache Kafka 3.7.0 Release

On an unrelated note, I found a blocker bug related to upgrades from 3.6 (and earlier) to 3.7. The JIRA is here: https://issues.apache.org/jira/browse/KAFKA-16094 Fix here: https://github.com/apache/kafka/pull/15153 best, Colin On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote: > Hi

Re: Apache Kafka 3.7.0 Release

Hi Ismael, I wasn't aware of that. If we are required to publish all modules, then this is working as intended. I am a bit curious if we've discussed why we need to publish the server modules to Sonatype. Is there a discussion about the pros and cons of this somewhere? regards, Colin On Mon,

Re: Apache Kafka 3.7.0 Release

All modules are published to Sonatype - that's a requirement. You may be missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`. Ismael On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe wrote: > Hi Ismael, > > It seems like both the metadata gradle module and the server-common

Re: Apache Kafka 3.7.0 Release

Hi Ismael, It seems like both the metadata gradle module and the server-common module are getting published to Sonatype as separate artifacts, unless I'm misunderstanding something. Example: https://central.sonatype.com/search?q=kafka-server-common I don't see kafka-core getting published,

Re: Apache Kafka 3.7.0 Release

Hi all, Are you talking about publishing the artefacts to maven central? Looking at the history [1], it seems that the metadata module has been published since we have it. I also see other internal modules there too. [1]

Re: Apache Kafka 3.7.0 Release

Hi Colin, I think you may have misunderstood what they mean by gradle metadata - it's not the Kafka metadata module. Ismael On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe wrote: > Oops, hit send too soon. I see that #15127 was already merged. So we > should no longer be publishing :metadata as

Re: Apache Kafka 3.7.0 Release

Oops, hit send too soon. I see that #15127 was already merged. So we should no longer be publishing :metadata as part of the clients artifacts, right? thanks, Colin On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote: > Hi Apporv, > > Please remove the metadata module from any artifacts

Re: Apache Kafka 3.7.0 Release

Hi Apporv, Please remove the metadata module from any artifacts published for clients. It is only used by the server. best, Colin On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote: > Hi Colin, > Thanks for the response. The only reason for asking the question of > publishing the metadata is

Re: Apache Kafka 3.7.0 Release

I've almost got an RC candidate out to test. I just need a. review on the kafka-site repo to update the website with the appropriate 3.7 subpages - https://github.com/apache/kafka-site/pull/576/ On Mon, Jan 8, 2024 at 10:05 AM Lucas Brutschy wrote: > Hi, > > we have fixed one memory leak in

Re: Apache Kafka 3.7.0 Release

Hi, we have fixed one memory leak in Kafka Streams, but there is still at least one missing in the code. I created https://issues.apache.org/jira/browse/KAFKA-16089 which is a blocker. Cheers, Lucas On Sun, Jan 7, 2024 at 12:05 PM Apoorv Mittal wrote: > > Hi Colin, > Thanks for the response.

Re: Apache Kafka 3.7.0 Release

Hi Colin, Thanks for the response. The only reason for asking the question of publishing the metadata is because that's present in previous client releases. For more context, the description of PR holds the details and waiting for the confirmation there

Re: Apache Kafka 3.7.0 Release

metadata is an internal gradle module. It is not used by clients. So I don't see why you would want to publish it (unless I'm misunderstanding something). best, Colin On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote: > Thanks for reporting the blockers, folks. Good job finding. > > I

Re: Apache Kafka 3.7.0 Release

Thanks for reporting the blockers, folks. Good job finding. I have one ask - can anybody with Gradle expertise help review this small PR? https://github.com/apache/kafka/pull/15127 (+1, -1) In particular, we are wondering whether we need to publish module metadata as part of the gradle publishing

Re: Apache Kafka 3.7.0 Release

We have potentially one more blocker https://issues.apache.org/jira/browse/KAFKA-16082 which might cause a data loss scenario with JBOD in KRaft. Initial analysis thought this is a problem and further review looks like it isn't but we are continuing to dig into the issue to ensure that it isn't.

Re: Apache Kafka 3.7.0 Release

One more blocker (memory leak) for 3.7: https://issues.apache.org/jira/browse/KAFKA-16086 -Matthias On 1/4/24 9:53 PM, Stanislav Kozlovski wrote: Thanks Apoorv, I was going to update the mailing thread as well. Major kudos to Apoorv for the thorough work debugging and getting to the bottom

Re: Apache Kafka 3.7.0 Release

Thanks Apoorv, I was going to update the mailing thread as well. Major kudos to Apoorv for the thorough work debugging and getting to the bottom of this tricky publishing issue, a subtle regression from the work he did in making the kafka-clients jar shadowed. On Thu, Jan 4, 2024 at 5:09 PM

Re: Apache Kafka 3.7.0 Release

Hi Stan, I have opened the minor PR: https://github.com/apache/kafka/pull/15127 to fix publishing the dependency. Once discussed and merged in trunk, I'll update the 3.7 branch as well. Regards, Apoorv Mittal +44 7721681581 On Thu, Jan 4, 2024 at 12:49 PM Matthias J. Sax wrote: > We found a

Re: Apache Kafka 3.7.0 Release

We found a blocker for 3.7: https://issues.apache.org/jira/browse/KAFKA-16077 Already having a PR under review to fix it. -Matthias On 1/3/24 10:43 AM, Stanislav Kozlovski wrote: Hey all, happy new year. Thanks for the heads up Almog. Makes sense. To give an update - I haven't been able

Re: Apache Kafka 3.7.0 Release

Hey all, happy new year. Thanks for the heads up Almog. Makes sense. To give an update - I haven't been able to resolve the gradlewAll publish failure, and as such haven't been able to release an RC. As a minor barrier, I have to also update the year in the NOTICE file, otherwise the release

Re: Apache Kafka 3.7.0 Release

Hello Stan, I wanted to give you a heads up that https://github.com/apache/kafka/pull/15073 ( https://issues.apache.org/jira/browse/KAFKA-16046) was identified as a blocker regression and should be merged to trunk by EOD. Cheers, Almog On Tue, Jan 2, 2024 at 4:20 AM Stanislav Kozlovski wrote:

Re: Apache Kafka 3.7.0 Release

Hi Apoorv, Thanks for taking ownership and looking into this! One more caveat is that I believe this first publish is ran with JDK 8, as the release.py runs with both JDK 8 and (if I recall correctly) 17 versions. This seems to fail on the first one - so JDK 8. Not sure if that is related in any

Re: Apache Kafka 3.7.0 Release

Just to update this thread, everything in KAFKA-14127 is done now. A few tasks got moved to a separate umbrella JIRA. Some folks are going to do more testing, both manual and automated, in the next week or two. I think this will give us a good indicator of stability and what we need to fix.

Re: Apache Kafka 3.7.0 Release

Hey all, Code Freeze has been over for some time, and I have been trying to build an RC. I have hit a problem trying to build it, hence am reaching out for some help. The error I'm getting while running release.py is: ./gradlewAll publish ... > * What went wrong: > Execution failed for task >

Re: Apache Kafka 3.7.0 Release

Hi, With the current timeline for 3.7, I tend to agree with Viktor that JBOD support in KRaft is unlikely to receive the extensive testing this feature needs before releasing. And that's not counting the testing tasks left to do in https://issues.apache.org/jira/browse/KAFKA-14127. I'm fine

Re: Apache Kafka 3.7.0 Release

Hi Viktor, Extending the code freeze doesn't help stabilize things. If we have important bugs for JBOD, we should mark those as blockers and we'll wait until they are fixed if the fixes won't take too long (as usual). Ismael On Tue, Dec 19, 2023 at 11:58 AM Viktor Somogyi-Vass wrote: > Hi

Re: Apache Kafka 3.7.0 Release

Hi all, I was wondering what people think about extending the code freeze date to early January? The reason I'm asking is that there are still a couple of testing gaps in JBOD (https://issues.apache.org/jira/browse/KAFKA-14127) which I think is very important to finish to ensure a high quality

Re: Apache Kafka 3.7.0 Release

Hey! Just notifying everybody on this thread that I have cut the 3.7 branch and sent a new email thread titled "New Release Branch 3.7" to the mailing list . Best, Stanislav On Wed, Dec 6, 2023 at 11:10 AM Stanislav Kozlovski

Re: Apache Kafka 3.7.0 Release

Hello again, Time is flying by! It is feature freeze day! By today, we expect to have major features merged and to begin working on their stabilisation. Minor features should have PRs. I am planning to cut the release branch soon - on Monday EU daytime. When I do that, I will create a new

Re: Apache Kafka 3.7.0 Release

Hey all, The KIP Freeze has passed. I count 31 KIPs that will be going into the 3.7 Release. Thank you all for your hard work! They are the following (some of these were accepted in previous releases and have minor parts going out, some targeting a Preview release and the rest being fully

Re: Apache Kafka 3.7.0 Release

Hi Stan, I'd like to propose including KIP-892 in the 3.7 release. The KIP has been accepted and I'm just working on rebasing the implementation against trunk before I open a PR. Regards, Nick On Tue, 21 Nov 2023 at 11:27, Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > Hi Stan > >

Re: Apache Kafka 3.7.0 Release

Hi Stan Can you include KIP-951 to the 3.7 release plan? All PRs are merged in the trunk. On Wed, Nov 15, 2023 at 4:05 PM Stanislav Kozlovski wrote: > Friendly reminder to everybody that the KIP Freeze is *exactly 7 days away* > - November 22. > > A KIP must be accepted by this date in order

Re: Apache Kafka 3.7.0 Release

Friendly reminder to everybody that the KIP Freeze is *exactly 7 days away* - November 22. A KIP must be accepted by this date in order to be considered for this release. Note, any KIP that may not be implemented in time, or otherwise risks heavily destabilizing the release, should be deferred.

Re: Apache Kafka 3.7.0 Release

Looks great, thank you! +1 On Thu, Nov 2, 2023 at 10:21 AM David Jacot wrote: > +1 from me as well. Thanks, Stan! > > David > > On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma wrote: > > > Thanks Stanislav, +1 > > > > Ismael > > > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski > > wrote: > >

Re: Apache Kafka 3.7.0 Release

This makes sense to me. Thanks for following up, Stan. On Thu, Nov 2, 2023 at 7:02 AM Stanislav Kozlovski wrote: > Hi all, > > Given the discussion here and the lack of any pushback, I have changed the > dates of the release: > - KIP Freeze - *November 22 *(moved 4 days later) > - Feature

Re: Apache Kafka 3.7.0 Release

+1 from me as well. Thanks, Stan! David On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma wrote: > Thanks Stanislav, +1 > > Ismael > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski > wrote: > > > Hi all, > > > > Given the discussion here and the lack of any pushback, I have changed > the > >

Re: Apache Kafka 3.7.0 Release

Thanks Stanislav, +1 Ismael On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski wrote: > Hi all, > > Given the discussion here and the lack of any pushback, I have changed the > dates of the release: > - KIP Freeze - *November 22 *(moved 4 days later) > - Feature Freeze - *December 6 *(moved 2

Re: Apache Kafka 3.7.0 Release

Hi all, Given the discussion here and the lack of any pushback, I have changed the dates of the release: - KIP Freeze - *November 22 *(moved 4 days later) - Feature Freeze - *December 6 *(moved 2 days earlier) - Code Freeze - *December 20* If anyone has any thoughts against this proposal -

Re: Apache Kafka 3.7.0 Release

Thanks for the response and explanations -- I think the main question for me was whether we intended to permanently increase the KF -- FF gap from the historical 1 week to 3 weeks? Maybe this was a conscious decision and I just missed the memo, hopefully someone else can chime in here. I'm all

Re: Apache Kafka 3.7.0 Release

Thanks for the thorough response, Sophie. - Added to the "Future Release Plan" > 1. Why is the KIP freeze deadline on a Saturday? It was simply added as a starting point - around 30 days from the announcement. We can move it earlier to the 15th of November, but my thinking is later is better

Re: Apache Kafka 3.7.0 Release

Actually I have a few questions about the schedule: 1. Why is the KIP freeze deadline on a Saturday? Traditionally this has been on a Wednesday, which is nice because it gives people until Monday to kick off the vote and give people a full 3 working days to review and vote on it. Also, 2. Why are

Re: Apache Kafka 3.7.0 Release

Can you add the 3.7 plan to the release schedule page? (this --> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan) Thanks! On Sun, Oct 15, 2023 at 2:27 AM Stanislav Kozlovski wrote: > Hey Chris, > > Thanks for the catch! It was indeed copied and I wasn't sure what to make

Re: Apache Kafka 3.7.0 Release

Hey Chris, Thanks for the catch! It was indeed copied and I wasn't sure what to make of the bullet point, so I kept it. What you say makes sense - I removed it. I also added KIP-976! Cheers! On Sat, Oct 14, 2023 at 9:35 PM Chris Egerton wrote: > Hi Stanislav, > > Thanks for putting this

Re: Apache Kafka 3.7.0 Release

Hi Stanislav, Thanks for putting this together! I think the "Ensure that release candidates include artifacts for the new Connect test-plugins module" section (which I'm guessing was copied over from the 3.6.0 release plan?) can be removed; we made sure that those artifacts were present for

Re: Apache Kafka 3.7.0 Release

Thanks for letting me drive it, folks. I've created the 3.7.0 release page here: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0 It outlines the key milestones and important dates for the release. In particular, since the last two releases slipped their originally targeted

Re: Apache Kafka 3.7.0 Release

Thanks Stan! +1 Best, Bruno On 10/10/23 7:24 AM, Luke Chen wrote: Thanks Stanislav! On Tue, Oct 10, 2023 at 3:05 AM Josep Prat wrote: Thanks Stanislav! ——— Josep Prat Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer:

Re: Apache Kafka 3.7.0 Release

Thanks Stanislav! On Tue, Oct 10, 2023 at 3:05 AM Josep Prat wrote: > Thanks Stanislav! > > ——— > Josep Prat > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491715557497 >

Re: Apache Kafka 3.7.0 Release

Thanks Stanislav! ——— Josep Prat Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen m: +491715557497 w: aiven.io e: josep.p...@aiven.io On Mon, Oct 9, 2023, 20:05 Chris Egerton wrote: > +1,

Re: Apache Kafka 3.7.0 Release

+1, thanks Stanislav! On Mon, Oct 9, 2023, 14:02 Bill Bejeck wrote: > +1 > > Thanks, Stanislav! > > -Bill > > On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma wrote: > > > Thanks for volunteering Stanislav! > > > > Ismael > > > > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski > > wrote: > > >

Re: Apache Kafka 3.7.0 Release

+1 Thanks, Stanislav! -Bill On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma wrote: > Thanks for volunteering Stanislav! > > Ismael > > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski > wrote: > > > Hey all! > > > > I would like to volunteer to be the release manager driving the next > >

Re: Apache Kafka 3.7.0 Release

Thanks for volunteering Stanislav! Ismael On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski wrote: > Hey all! > > I would like to volunteer to be the release manager driving the next > release - Apache Kafka *3.7.0*. > > If there are no objections, I will start and share a release plan soon

Apache Kafka 3.7.0 Release

Hey all! I would like to volunteer to be the release manager driving the next release - Apache Kafka *3.7.0*. If there are no objections, I will start and share a release plan soon enough! Cheers, Stanislav