[jira] [Resolved] (KAFKA-12819) Quality of life improvements for tests

[Konstantine Karantasis (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-12819.

Resolution: Fixed

> Quality of life improvements for tests
> --
>
> Key: KAFKA-12819
> URL: https://issues.apache.org/jira/browse/KAFKA-12819
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Matthew de Detrich
>Assignee: Matthew de Detrich
>Priority: Minor
> Fix For: 3.0.0
>
>
> Minor improvements to various tests, such as using assertObject instead of 
> assertEquals (when comparing objects), fill in missing messages in asserts 
> etc etc



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Israel Ekpo]

+1 on the new schedule.

On Wed, May 26, 2021 at 8:14 PM Sophie Blee-Goldman
 wrote:

> Ah ok, thanks Konstantine. I won't bug you about every new KIP that comes
> in between now and KIP Freeze :P
>
> +1 on the scheduling changes as well
>
> On Wed, May 26, 2021 at 4:00 PM David Arthur  wrote:
>
> > The new schedule looks good to me, +1
> >
> > On Wed, May 26, 2021 at 6:29 PM Ismael Juma  wrote:
> >
> > > Thanks Konstantine, +1 from me.
> > >
> > > Ismael
> > >
> > > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Please find below the updated release plan for the Apache Kafka 3.0.0
> > > > release.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > >
> > > > New suggested dates for the release are as follows:
> > > >
> > > > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > > > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > > >
> > > > At least two weeks of stabilization will follow Code Freeze.
> > > >
> > > > The release plan is up to date and currently includes all the
> approved
> > > KIPs
> > > > that are targeting 3.0.0.
> > > >
> > > > Please let me know if you have any objections with the recent
> extension
> > > of
> > > > Feature Freeze and Code Freeze or any other concerns.
> > > >
> > > > Regards,
> > > > Konstantine
> > > >
> > >
> > --
> > David Arthur
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #166

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Ismael Juma]

Perfect, thanks.

On Wed, May 26, 2021, 2:43 PM Konstantine Karantasis
 wrote:

> Thanks for bringing up this suggestion, Ismael.
>
> Given that the date for adopting new features (aka KIP Freeze) remains the
> same and that this release is a major release with goals such as the ones
> mentioned by Israel I'm also open to and in agreement with the extension of
> Feature Freeze, Code Freeze and the beginning of stabilization phase by two
> weeks respectively.
>
> To make this suggestion more apparent and give everyone the opportunity to
> raise and discuss any concerns, I'll open a new thread for the 3.0.0 Apache
> Kafka release that includes the new release dates.
>
> Let's take any future discussions with respect to the 3.0.0 release plan in
> that new thread if you agree.
>
> Best,
> Konstantine
>
>
> On Wed, May 26, 2021 at 8:45 AM Colin McCabe  wrote:
>
> > On Wed, May 26, 2021, at 07:18, Ismael Juma wrote:
> > > Hi Konstantine,
> > >
> > > Looking at the schedule and some of the ongoing work in KIP-500 (one of
> > the
> > > KIPs that is important to land in 3.0), I think we'll need a bit more
> > time.
> > > We definitely do not want to cause a disruption to our time-based
> release
> > > process, but given that this is a major release, I think a small
> > adjustment
> > > would make sense. I suggest we move the feature freeze and code freeze
> > > dates by 2 weeks and leave the KIP freeze as it is:
> > >
> > > KIP Freeze: 09 Jun 2021
> > > Feature Freeze: 30 Jun 2021
> > > Code Freeze: 14 July 2021
> > >
> > > This would ensure we have a solid 3.0 release. Thoughts?
> > >
> >
> > +1 for the proposed dates. Thanks, Ismael.
> >
> > best,
> > Colin
> >
> > >
> > > On Tue, May 25, 2021 at 6:40 AM Dongjin Lee 
> wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > Please Add the following KIPs into Kafka 3.0 plan. In the case of
> > KIP-653,
> > > > it was passed already but not included in the 2.8.0 release.
> > > >
> > > > - KIP-653: Upgrade log4j to log4j2
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > > > >
> > > > - KIP-719: Add Log4J2 Appender
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > > > >
> > > >
> > > > I also updated the proposal documents reflecting the recent changes
> to
> > our
> > > > codebase. Especially:
> > > >
> > > > 1. KIP-653 document
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > > > >
> > > > now explains which modules will be migrated and why.
> > > > 2. KIP-719 document
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > > > >
> > > > now explains not only the log4j2-appender plan but also upgrading the
> > > > omitted modules in KIP-653 into log4j2.
> > > >
> > > > As you can see here, those two KIPs are the different parts of the
> same
> > > > problem. I believe the community will have a good grasp on why both
> > KIPs
> > > > are best if released altogether.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
> > > >  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Lots of good progress recently towards the 3.0.0 release and given
> > that
> > > > we
> > > > > are now a little less than a month away from the first milestone, I
> > > > wanted
> > > > > to post a status update and a reminder of the upcoming dates for
> this
> > > > major
> > > > > release.
> > > > >
> > > > > According to the current release plan for Apache Kafka 3.0.0:
> > > > >
> > > > > KIP Freeze is 09 Jun 2021
> > > > >
> > > > > Feature Freeze is 16 Jun 2021
> > > > >
> > > > > Code Freeze is 30 Jun 2021
> > > > >
> > > > > and at least two weeks of stabilization will follow Code Freeze.
> > > > >
> > > > > Since the initial announcement, the list of adopted KIPs targeting
> > 3.0.0
> > > > > has grown with 10 additional KIPs. Several more are under
> discussion
> > and
> > > > > voting.
> > > > >
> > > > > New KIPs will be accepted up until the KIP Freeze and of course the
> > > > actual
> > > > > set of features that will make it to 3.0.0 will be finalized right
> > after
> > > > > Feature Freeze.
> > > > >
> > > > > With that in mind, if you want to assist the 3.0.0 release process,
> > > > please
> > > > > make sure that the adopted KIPs which you aim to include in this
> > major
> > > > > release have the right number in the “Release” column of the table
> > in the
> > > > > KIP wiki and that their respective Jira tickets include 3.0.0 in
> the
> > "Fix
> > > > > Version/s" label.
> > > > >
> > > > > Kafka Improvement Proposals:
> > > > >
> > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > >
> > > > > Release Plan 3.0.0:
> > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > > > >
>

[GitHub] [kafka-site] prince-mahajan commented on pull request #356: Fix broken link

[GitBox]

prince-mahajan commented on pull request #356:
URL: https://github.com/apache/kafka-site/pull/356#issuecomment-84920


   > Thanks @prince-mahajan
   
   Thanks! Can you also approve this please? 
https://github.com/apache/kafka/pull/10771 (same change in a different place).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Sophie Blee-Goldman]

Ah ok, thanks Konstantine. I won't bug you about every new KIP that comes
in between now and KIP Freeze :P

+1 on the scheduling changes as well

On Wed, May 26, 2021 at 4:00 PM David Arthur  wrote:

> The new schedule looks good to me, +1
>
> On Wed, May 26, 2021 at 6:29 PM Ismael Juma  wrote:
>
> > Thanks Konstantine, +1 from me.
> >
> > Ismael
> >
> > On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
> >  wrote:
> >
> > > Hi all,
> > >
> > > Please find below the updated release plan for the Apache Kafka 3.0.0
> > > release.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > >
> > > New suggested dates for the release are as follows:
> > >
> > > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> > >
> > > At least two weeks of stabilization will follow Code Freeze.
> > >
> > > The release plan is up to date and currently includes all the approved
> > KIPs
> > > that are targeting 3.0.0.
> > >
> > > Please let me know if you have any objections with the recent extension
> > of
> > > Feature Freeze and Code Freeze or any other concerns.
> > >
> > > Regards,
> > > Konstantine
> > >
> >
> --
> David Arthur
>

[GitHub] [kafka-site] prince-mahajan opened a new pull request #356: Fix broken link

[GitBox]

prince-mahajan opened a new pull request #356:
URL: https://github.com/apache/kafka-site/pull/356


   Fixing the link to a cited blog. The existing link now points to a steroid 
website so need to pull the blog from internet archives.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Permission to create KIPs

[Guozhang Wang]

Hello Josep,

I saw your id "josep.prat" is already added on the wiki, you should be able
to create KIPs now.

On Wed, May 26, 2021 at 10:09 AM Josep Prat 
wrote:

> Hi there,
>
> I would like to have permissions to create KIPs on the wiki.
> My username should be josep.prat.
>
> Thanks,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>


-- 
-- Guozhang

[GitHub] [kafka-site] guozhangwang commented on pull request #356: Fix broken link

[GitBox]

guozhangwang commented on pull request #356:
URL: https://github.com/apache/kafka-site/pull/356#issuecomment-849183797


   Thanks @prince-mahajan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] guozhangwang merged pull request #356: Fix broken link

[GitBox]

guozhangwang merged pull request #356:
URL: https://github.com/apache/kafka-site/pull/356


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[David Arthur]

The new schedule looks good to me, +1

On Wed, May 26, 2021 at 6:29 PM Ismael Juma  wrote:

> Thanks Konstantine, +1 from me.
>
> Ismael
>
> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>  wrote:
>
> > Hi all,
> >
> > Please find below the updated release plan for the Apache Kafka 3.0.0
> > release.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >
> > New suggested dates for the release are as follows:
> >
> > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> >
> > At least two weeks of stabilization will follow Code Freeze.
> >
> > The release plan is up to date and currently includes all the approved
> KIPs
> > that are targeting 3.0.0.
> >
> > Please let me know if you have any objections with the recent extension
> of
> > Feature Freeze and Code Freeze or any other concerns.
> >
> > Regards,
> > Konstantine
> >
>
-- 
David Arthur

Re: EXTERNAL: Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Guozhang Wang]

Hi Meiling,

Could you double check if the email sent is from non-sanboxed "
m...@rockwellautomation.com" instead of "m...@rockwellautomation.com.invalid"?
I believe the dev-unsubscr...@kafka.apache.org does work.

On Wed, May 26, 2021 at 3:32 PM Meiling He
 wrote:

> Can anyone instruct me how to unsubscribe from this mail list, please!!
> dev-unsubscr...@kafka.apache.org doesn’t work.
>
> 
> From: Ismael Juma 
> Sent: Wednesday, May 26, 2021 5:29:03 PM
> To: dev 
> Subject: EXTERNAL: Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new
> updated dates
>
> [Use caution with links & attachments]
>
>
>
> Thanks Konstantine, +1 from me.
>
> Ismael
>
> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>  wrote:
>
> > Hi all,
> >
> > Please find below the updated release plan for the Apache Kafka 3.0.0
> > release.
> >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466__;!!JhrIYaSK6lFZ!-k0g-tIlibi6VoGNY2-ZLKPCfAz2RJ2ICdx4IA4TfulZQIp7Gx7rhD9f0KyCN4Gh2VwyHw$
> >
> > New suggested dates for the release are as follows:
> >
> > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> >
> > At least two weeks of stabilization will follow Code Freeze.
> >
> > The release plan is up to date and currently includes all the approved
> KIPs
> > that are targeting 3.0.0.
> >
> > Please let me know if you have any objections with the recent extension
> of
> > Feature Freeze and Code Freeze or any other concerns.
> >
> > Regards,
> > Konstantine
> >
>


-- 
-- Guozhang

Re: EXTERNAL: Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Meiling He]

Can anyone instruct me how to unsubscribe from this mail list, please!!  
dev-unsubscr...@kafka.apache.org doesn’t work.


From: Ismael Juma 
Sent: Wednesday, May 26, 2021 5:29:03 PM
To: dev 
Subject: EXTERNAL: Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new 
updated dates

[Use caution with links & attachments]



Thanks Konstantine, +1 from me.

Ismael

On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
 wrote:

> Hi all,
>
> Please find below the updated release plan for the Apache Kafka 3.0.0
> release.
>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466__;!!JhrIYaSK6lFZ!-k0g-tIlibi6VoGNY2-ZLKPCfAz2RJ2ICdx4IA4TfulZQIp7Gx7rhD9f0KyCN4Gh2VwyHw$
>
> New suggested dates for the release are as follows:
>
> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> Code Freeze is 14 July 2021 (new date, extended by two weeks)
>
> At least two weeks of stabilization will follow Code Freeze.
>
> The release plan is up to date and currently includes all the approved KIPs
> that are targeting 3.0.0.
>
> Please let me know if you have any objections with the recent extension of
> Feature Freeze and Code Freeze or any other concerns.
>
> Regards,
> Konstantine
>

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Ismael Juma]

Thanks Konstantine, +1 from me.

Ismael

On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
 wrote:

> Hi all,
>
> Please find below the updated release plan for the Apache Kafka 3.0.0
> release.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>
> New suggested dates for the release are as follows:
>
> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> Code Freeze is 14 July 2021 (new date, extended by two weeks)
>
> At least two weeks of stabilization will follow Code Freeze.
>
> The release plan is up to date and currently includes all the approved KIPs
> that are targeting 3.0.0.
>
> Please let me know if you have any objections with the recent extension of
> Feature Freeze and Code Freeze or any other concerns.
>
> Regards,
> Konstantine
>

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Konstantine Karantasis]

Thanks Sophie.

I'm also tracking the updates on the main KIP page and the Adopted KIPs
table periodically, moving newly adopted KIPs to the release plan.
But we'll definitely use the mailing list more as we approach KIP Freeze
and make sure we include everything that is ready by the time we hit
Feature Freeze.

Thanks for letting me know!
Konstantine


On Wed, May 26, 2021 at 3:04 PM Sophie Blee-Goldman
 wrote:

> Hey Konstantine, I did a quick skim over the Streams KIPs and found two
> more which had not
> been moved to the "Adopted" section on the main KIP page and are missing
> from the release
> notes. These are:
>
>- KIP-466: Add support for List serialization and deserialization
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization
> >
>- KIP-725: Streamlining configurations for WindowedSerializer and
>WindowedDeserializer
><
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047930
> >
>
>
> Thanks!
>  -Sophie
>
>
> On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
>  wrote:
>
> > Hi all,
> >
> > Please find below the updated release plan for the Apache Kafka 3.0.0
> > release.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> >
> > New suggested dates for the release are as follows:
> >
> > KIP Freeze is 09 June 2021 (same date as in the initial plan)
> > Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> > Code Freeze is 14 July 2021 (new date, extended by two weeks)
> >
> > At least two weeks of stabilization will follow Code Freeze.
> >
> > The release plan is up to date and currently includes all the approved
> KIPs
> > that are targeting 3.0.0.
> >
> > Please let me know if you have any objections with the recent extension
> of
> > Feature Freeze and Code Freeze or any other concerns.
> >
> > Regards,
> > Konstantine
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #165

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Sophie Blee-Goldman]

Hey Konstantine, I did a quick skim over the Streams KIPs and found two
more which had not
been moved to the "Adopted" section on the main KIP page and are missing
from the release
notes. These are:

   - KIP-466: Add support for List serialization and deserialization
   

   - KIP-725: Streamlining configurations for WindowedSerializer and
   WindowedDeserializer
   


Thanks!
 -Sophie


On Wed, May 26, 2021 at 2:48 PM Konstantine Karantasis
 wrote:

> Hi all,
>
> Please find below the updated release plan for the Apache Kafka 3.0.0
> release.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
>
> New suggested dates for the release are as follows:
>
> KIP Freeze is 09 June 2021 (same date as in the initial plan)
> Feature Freeze is 30 June 2021 (new date, extended by two weeks)
> Code Freeze is 14 July 2021 (new date, extended by two weeks)
>
> At least two weeks of stabilization will follow Code Freeze.
>
> The release plan is up to date and currently includes all the approved KIPs
> that are targeting 3.0.0.
>
> Please let me know if you have any objections with the recent extension of
> Feature Freeze and Code Freeze or any other concerns.
>
> Regards,
> Konstantine
>

[DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Konstantine Karantasis]

Hi all,

Please find below the updated release plan for the Apache Kafka 3.0.0
release.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466

New suggested dates for the release are as follows:

KIP Freeze is 09 June 2021 (same date as in the initial plan)
Feature Freeze is 30 June 2021 (new date, extended by two weeks)
Code Freeze is 14 July 2021 (new date, extended by two weeks)

At least two weeks of stabilization will follow Code Freeze.

The release plan is up to date and currently includes all the approved KIPs
that are targeting 3.0.0.

Please let me know if you have any objections with the recent extension of
Feature Freeze and Code Freeze or any other concerns.

Regards,
Konstantine

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Konstantine Karantasis]

Thanks for bringing up this suggestion, Ismael.

Given that the date for adopting new features (aka KIP Freeze) remains the
same and that this release is a major release with goals such as the ones
mentioned by Israel I'm also open to and in agreement with the extension of
Feature Freeze, Code Freeze and the beginning of stabilization phase by two
weeks respectively.

To make this suggestion more apparent and give everyone the opportunity to
raise and discuss any concerns, I'll open a new thread for the 3.0.0 Apache
Kafka release that includes the new release dates.

Let's take any future discussions with respect to the 3.0.0 release plan in
that new thread if you agree.

Best,
Konstantine


On Wed, May 26, 2021 at 8:45 AM Colin McCabe  wrote:

> On Wed, May 26, 2021, at 07:18, Ismael Juma wrote:
> > Hi Konstantine,
> >
> > Looking at the schedule and some of the ongoing work in KIP-500 (one of
> the
> > KIPs that is important to land in 3.0), I think we'll need a bit more
> time.
> > We definitely do not want to cause a disruption to our time-based release
> > process, but given that this is a major release, I think a small
> adjustment
> > would make sense. I suggest we move the feature freeze and code freeze
> > dates by 2 weeks and leave the KIP freeze as it is:
> >
> > KIP Freeze: 09 Jun 2021
> > Feature Freeze: 30 Jun 2021
> > Code Freeze: 14 July 2021
> >
> > This would ensure we have a solid 3.0 release. Thoughts?
> >
>
> +1 for the proposed dates. Thanks, Ismael.
>
> best,
> Colin
>
> >
> > On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:
> >
> > > Hi Konstantine,
> > >
> > > Please Add the following KIPs into Kafka 3.0 plan. In the case of
> KIP-653,
> > > it was passed already but not included in the 2.8.0 release.
> > >
> > > - KIP-653: Upgrade log4j to log4j2
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > > >
> > > - KIP-719: Add Log4J2 Appender
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > > >
> > >
> > > I also updated the proposal documents reflecting the recent changes to
> our
> > > codebase. Especially:
> > >
> > > 1. KIP-653 document
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > > >
> > > now explains which modules will be migrated and why.
> > > 2. KIP-719 document
> > > <
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > > >
> > > now explains not only the log4j2-appender plan but also upgrading the
> > > omitted modules in KIP-653 into log4j2.
> > >
> > > As you can see here, those two KIPs are the different parts of the same
> > > problem. I believe the community will have a good grasp on why both
> KIPs
> > > are best if released altogether.
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
> > >  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Lots of good progress recently towards the 3.0.0 release and given
> that
> > > we
> > > > are now a little less than a month away from the first milestone, I
> > > wanted
> > > > to post a status update and a reminder of the upcoming dates for this
> > > major
> > > > release.
> > > >
> > > > According to the current release plan for Apache Kafka 3.0.0:
> > > >
> > > > KIP Freeze is 09 Jun 2021
> > > >
> > > > Feature Freeze is 16 Jun 2021
> > > >
> > > > Code Freeze is 30 Jun 2021
> > > >
> > > > and at least two weeks of stabilization will follow Code Freeze.
> > > >
> > > > Since the initial announcement, the list of adopted KIPs targeting
> 3.0.0
> > > > has grown with 10 additional KIPs. Several more are under discussion
> and
> > > > voting.
> > > >
> > > > New KIPs will be accepted up until the KIP Freeze and of course the
> > > actual
> > > > set of features that will make it to 3.0.0 will be finalized right
> after
> > > > Feature Freeze.
> > > >
> > > > With that in mind, if you want to assist the 3.0.0 release process,
> > > please
> > > > make sure that the adopted KIPs which you aim to include in this
> major
> > > > release have the right number in the “Release” column of the table
> in the
> > > > KIP wiki and that their respective Jira tickets include 3.0.0 in the
> "Fix
> > > > Version/s" label.
> > > >
> > > > Kafka Improvement Proposals:
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > >
> > > > Release Plan 3.0.0:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > > >
> > > > Regards,
> > > >
> > > > Konstantine
> > > >
> > > >
> > > > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > > > konstant...@confluent.io> wrote:
> > > >
> > > > >
> > > > > Thank you and hi again.
> > > > >
> > > > > I just published a release plan at:
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > > >
> > > > > I

[jira] [Created] (KAFKA-12854) Add a config to allow skipping metadata cache update when topic partition is unassigned

[Lincong Li (Jira)]

Lincong Li created KAFKA-12854:
--

 Summary: Add a config to allow skipping metadata cache update when 
topic partition is unassigned
 Key: KAFKA-12854
 URL: https://issues.apache.org/jira/browse/KAFKA-12854
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Lincong Li


The "assign" method in the consumer triggers a metadata cache update if the new 
partition assignment is different from the current assignment. It makes sense 
to update the MD cache if the new assignment contains partitions that do not 
exist in the current assignment.

However, I wonder why is updating its MD cache necessary if the new partition 
assignment is a subset of the current assignment. For example, the new 
assignment is tp0, tp1, and the current assignment is tp0, tp1, tp2.

The current behavior does too many drawbacks in most cases. However, if the 
number of consumer instances is large and each consumer instance is constantly 
getting some topic partitions unassigned, the QPS of the MD request sent out to 
update the MD cache becomes high as a result.
 
Proposed changes:
Add a config to allow skipping metadata cache update when topic partition(s) is 
unassigned

Existing PR to a forked repo:
https://github.com/linkedin/kafka/pull/166/files



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12853) Implement broker-side KRaft snapshots

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-12853:


 Summary: Implement broker-side KRaft snapshots
 Key: KAFKA-12853
 URL: https://issues.apache.org/jira/browse/KAFKA-12853
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Colin McCabe
Assignee: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12852) Log.splitOverflowedSegment() doesn't generate producer snapshot for new segment

[Jun Rao (Jira)]

Jun Rao created KAFKA-12852:
---

 Summary: Log.splitOverflowedSegment() doesn't generate producer 
snapshot for new segment
 Key: KAFKA-12852
 URL: https://issues.apache.org/jira/browse/KAFKA-12852
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.0.0
Reporter: Jun Rao


Currently, we expect every segment (except the very first one) to have a 
corresponding producer snapshot file.  However, in 
Log.splitOverflowedSegment(), we don't seem to create producerSnapshot for the 
split segments with a new based offset. This may cause issue with tier storage 
since we expect each archived segment to have a producer snapshot.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #164

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12851) Flaky Test RaftEventSimulationTest.canMakeProgressIfMajorityIsReachable

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-12851:
--

 Summary: Flaky Test 
RaftEventSimulationTest.canMakeProgressIfMajorityIsReachable
 Key: KAFKA-12851
 URL: https://issues.apache.org/jira/browse/KAFKA-12851
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: A. Sophie Blee-Goldman
 Fix For: 3.0.0


Failed twice on a [PR 
build|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-10755/6/testReport/]
h3. Stacktrace

org.opentest4j.AssertionFailedError: expected:  but was:  at 
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55) at 
org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40) at 
org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:35) at 
org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:162) at 
org.apache.kafka.raft.RaftEventSimulationTest.canMakeProgressIfMajorityIsReachable(RaftEventSimulationTest.java:263)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Permission to create KIPs

[Josep Prat]

Hi there,

I would like to have permissions to create KIPs on the wiki.
My username should be josep.prat.

Thanks,

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Colin McCabe]

On Wed, May 26, 2021, at 07:18, Ismael Juma wrote:
> Hi Konstantine,
> 
> Looking at the schedule and some of the ongoing work in KIP-500 (one of the
> KIPs that is important to land in 3.0), I think we'll need a bit more time.
> We definitely do not want to cause a disruption to our time-based release
> process, but given that this is a major release, I think a small adjustment
> would make sense. I suggest we move the feature freeze and code freeze
> dates by 2 weeks and leave the KIP freeze as it is:
> 
> KIP Freeze: 09 Jun 2021
> Feature Freeze: 30 Jun 2021
> Code Freeze: 14 July 2021
> 
> This would ensure we have a solid 3.0 release. Thoughts?
> 

+1 for the proposed dates. Thanks, Ismael.

best,
Colin

> 
> On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:
> 
> > Hi Konstantine,
> >
> > Please Add the following KIPs into Kafka 3.0 plan. In the case of KIP-653,
> > it was passed already but not included in the 2.8.0 release.
> >
> > - KIP-653: Upgrade log4j to log4j2
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > - KIP-719: Add Log4J2 Appender
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> >
> > I also updated the proposal documents reflecting the recent changes to our
> > codebase. Especially:
> >
> > 1. KIP-653 document
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > now explains which modules will be migrated and why.
> > 2. KIP-719 document
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> > now explains not only the log4j2-appender plan but also upgrading the
> > omitted modules in KIP-653 into log4j2.
> >
> > As you can see here, those two KIPs are the different parts of the same
> > problem. I believe the community will have a good grasp on why both KIPs
> > are best if released altogether.
> >
> > Best,
> > Dongjin
> >
> > On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
> >  wrote:
> >
> > > Hi all,
> > >
> > > Lots of good progress recently towards the 3.0.0 release and given that
> > we
> > > are now a little less than a month away from the first milestone, I
> > wanted
> > > to post a status update and a reminder of the upcoming dates for this
> > major
> > > release.
> > >
> > > According to the current release plan for Apache Kafka 3.0.0:
> > >
> > > KIP Freeze is 09 Jun 2021
> > >
> > > Feature Freeze is 16 Jun 2021
> > >
> > > Code Freeze is 30 Jun 2021
> > >
> > > and at least two weeks of stabilization will follow Code Freeze.
> > >
> > > Since the initial announcement, the list of adopted KIPs targeting 3.0.0
> > > has grown with 10 additional KIPs. Several more are under discussion and
> > > voting.
> > >
> > > New KIPs will be accepted up until the KIP Freeze and of course the
> > actual
> > > set of features that will make it to 3.0.0 will be finalized right after
> > > Feature Freeze.
> > >
> > > With that in mind, if you want to assist the 3.0.0 release process,
> > please
> > > make sure that the adopted KIPs which you aim to include in this major
> > > release have the right number in the “Release” column of the table in the
> > > KIP wiki and that their respective Jira tickets include 3.0.0 in the "Fix
> > > Version/s" label.
> > >
> > > Kafka Improvement Proposals:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > Release Plan 3.0.0:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > >
> > > Regards,
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > >
> > > > Thank you and hi again.
> > > >
> > > > I just published a release plan at:
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > >
> > > > I have included all the currently approved KIPs. I'm expecting this
> > list
> > > > to grow as we approach KIP freeze.
> > > >
> > > > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> > > >
> > > > Please let me know if you have any objections.
> > > >
> > > > Regards,
> > > > Konstantine
> > > >
> > > >
> > > > On Wed, Feb 24, 2021 at 9:45 AM Chia-Ping Tsai 
> > > > wrote:
> > > >
> > > >> Thanks for taking over this hard job! +1
> > > >>
> > > >> On 2021/02/23 08:02:09, Konstantine Karantasis <
> > > konstant...@confluent.io>
> > > >> wrote:
> > > >> > Hi all,
> > > >> >
> > > >> > Given that we seem to reach an agreement that the feature release
> > > after
> > > >> the
> > > >> > upcoming 2.8.0 will be 3.0.0, I'd like to volunteer to be the
> > release
> > > >> > manager for the Apache Kafka 3.0.0 release.
> > > >> >
> > > >> > It's a major release, so I thought it'd be helpful to start
> > planning a
> > > >> bit
> > > >> > in advance.
> > > >> >
> > > >> > If there are no objections, I'll start working o

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Josep Prat]

Hi Ryanne,
Apologies to bring this here then! :)

On Wed, May 26, 2021 at 5:27 PM Ryanne Dolan  wrote:

> Josep, that is being done as KIP-707. Looking forward to that as well :)
>
> Ryanne
>
> On Wed, May 26, 2021, 9:08 AM Josep Prat 
> wrote:
>
> > Sorry, I meant `CompletionStage` (instead of CompletableFuture) as this
> is
> > the interface.
> >
> > Best,
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Wed, May 26, 2021, 15:36 Josep Prat  wrote:
> >
> > > Hi,
> > > If I may, I would like to suggest that instead of using Java's `Future`
> > > class on the API's, it would be better to use `CompletableFuture`. This
> > > would offer the possibility of applying callbacks on its completion for
> > > example.
> > >
> > > Best,
> > >
> > > On Wed, May 26, 2021 at 3:28 PM Matthew de Detrich
> > >  wrote:
> > >
> > >> Maybe there was a miscommunication but I agree with everything you
> > said, I
> > >> was just clarifying what my definition of blocking is (because I think
> > >> there was a misunderstanding).
> > >>
> > >> And yes you are right, there is a limited amount of threads which is
> why
> > >> blocking is a bad thing because having threads sitting around
> > waiting/not
> > >> doing anything is a waste of resources but ultimately this is also a
> > >> performance problem because if you don't block you can simply process
> > more
> > >> IO tasks on a given machine/instance (hence greater performance).
> > >>
> > >> In any case, as is clarified the current behavior of send() needs to
> be
> > >> fixed. It's returning a Future but since it's internally blocking and
> > >> using
> > >> the caller's thread from an API perspective it gives the incorrect
> > >> impression that it's asynchronous (when it's not).
> > >>
> > >> On Wed, May 26, 2021 at 3:15 PM Ryanne Dolan 
> > >> wrote:
> > >>
> > >> > Matthew, it's more than performance tho. In many frameworks the
> number
> > >> of
> > >> > request threads is purposefully constrained, and blocking one means
> > you
> > >> > have one less to handle requests with. When you're handling a large
> > >> amount
> > >> > of requests with a small number of threads, any blocking can lead to
> > >> thread
> > >> > exhaustion.
> > >> >
> > >> > For this reason, you'll often see send() wrapped in a future or
> thread
> > >> > pool. But it's surprising that this would be required, since send()
> > >> already
> > >> > returns a future.
> > >> >
> > >> > Additionally, even when send() does not actually block, it does a
> lot
> > of
> > >> > work on the caller's thread, which is likewise surprising given a
> > >> future is
> > >> > returned. The effect is the same: less threads are available to
> handle
> > >> > requests, and you risk thread exhaustion.
> > >> >
> > >> > I think we may incidentally improve performance if we introduce an
> > >> internal
> > >> > thread pool, but the primary motivation here, at least for me, is to
> > fix
> > >> > the lie the API is telling, not to improve performance.
> > >> >
> > >> > Ryanne
> > >> >
> > >> >
> > >> >
> > >> > On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
> > >> >  wrote:
> > >> >
> > >> > > I think we may need to clarify terminology here, at least to me
> > >> blocking
> > >> > > means suspending a current thread to wait for some operation
> (which
> > is
> > >> > > wasteful if we are dealing with IO bound tasks). In other words,
> the
> > >> > > "blocking" is an implementation detail on how to wait rather than
> > >> whether
> > >> > > we need to wait or not, so to me this is more of a performance
> > >> question.
> > >> > >
> > >> > > In the scenario you describe of kafka clients producing too many
> > >> > messages,
> > >> > > as you said buffering is what should be done but I wouldn't
> classify
> > >> this
> > >> > > as blocking.
> > >> > >
> > >> > > On Mon, May 24, 2021 at 7:54 PM Colin McCabe 
> > >> wrote:
> > >> > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > I agree that we should give users the option of having a fully
> > async
> > >> > API,
> > >> > > > but I don't think external thread pools or queues are the right
> > >> > direction
> > >> > > > to go here. They add performance overheads and don't address the
> > >> root
> > >> > > > causes of the problem.
> > >> > > >
> > >> > > > There are basically two scenarios where we block, currently. One
> > is
> > >> > when
> > >> > > > we are doing a metadata fetch. I think this is clearly a bug, or
> > at
> > >> > least
> > >> > > > an implementation limitation. From the user's point of view, the
> > >> fact
> > >> > > that
> > >> > > > we are doing a metadata fetch is an implementation detail that
> > >> really
> > >> > > > shouldn't be exposed like this. We have talked about fixing this
> > in
> > >> the
> > >> > > >

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Ryanne Dolan]

Josep, that is being done as KIP-707. Looking forward to that as well :)

Ryanne

On Wed, May 26, 2021, 9:08 AM Josep Prat 
wrote:

> Sorry, I meant `CompletionStage` (instead of CompletableFuture) as this is
> the interface.
>
> Best,
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Wed, May 26, 2021, 15:36 Josep Prat  wrote:
>
> > Hi,
> > If I may, I would like to suggest that instead of using Java's `Future`
> > class on the API's, it would be better to use `CompletableFuture`. This
> > would offer the possibility of applying callbacks on its completion for
> > example.
> >
> > Best,
> >
> > On Wed, May 26, 2021 at 3:28 PM Matthew de Detrich
> >  wrote:
> >
> >> Maybe there was a miscommunication but I agree with everything you
> said, I
> >> was just clarifying what my definition of blocking is (because I think
> >> there was a misunderstanding).
> >>
> >> And yes you are right, there is a limited amount of threads which is why
> >> blocking is a bad thing because having threads sitting around
> waiting/not
> >> doing anything is a waste of resources but ultimately this is also a
> >> performance problem because if you don't block you can simply process
> more
> >> IO tasks on a given machine/instance (hence greater performance).
> >>
> >> In any case, as is clarified the current behavior of send() needs to be
> >> fixed. It's returning a Future but since it's internally blocking and
> >> using
> >> the caller's thread from an API perspective it gives the incorrect
> >> impression that it's asynchronous (when it's not).
> >>
> >> On Wed, May 26, 2021 at 3:15 PM Ryanne Dolan 
> >> wrote:
> >>
> >> > Matthew, it's more than performance tho. In many frameworks the number
> >> of
> >> > request threads is purposefully constrained, and blocking one means
> you
> >> > have one less to handle requests with. When you're handling a large
> >> amount
> >> > of requests with a small number of threads, any blocking can lead to
> >> thread
> >> > exhaustion.
> >> >
> >> > For this reason, you'll often see send() wrapped in a future or thread
> >> > pool. But it's surprising that this would be required, since send()
> >> already
> >> > returns a future.
> >> >
> >> > Additionally, even when send() does not actually block, it does a lot
> of
> >> > work on the caller's thread, which is likewise surprising given a
> >> future is
> >> > returned. The effect is the same: less threads are available to handle
> >> > requests, and you risk thread exhaustion.
> >> >
> >> > I think we may incidentally improve performance if we introduce an
> >> internal
> >> > thread pool, but the primary motivation here, at least for me, is to
> fix
> >> > the lie the API is telling, not to improve performance.
> >> >
> >> > Ryanne
> >> >
> >> >
> >> >
> >> > On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
> >> >  wrote:
> >> >
> >> > > I think we may need to clarify terminology here, at least to me
> >> blocking
> >> > > means suspending a current thread to wait for some operation (which
> is
> >> > > wasteful if we are dealing with IO bound tasks). In other words, the
> >> > > "blocking" is an implementation detail on how to wait rather than
> >> whether
> >> > > we need to wait or not, so to me this is more of a performance
> >> question.
> >> > >
> >> > > In the scenario you describe of kafka clients producing too many
> >> > messages,
> >> > > as you said buffering is what should be done but I wouldn't classify
> >> this
> >> > > as blocking.
> >> > >
> >> > > On Mon, May 24, 2021 at 7:54 PM Colin McCabe 
> >> wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > I agree that we should give users the option of having a fully
> async
> >> > API,
> >> > > > but I don't think external thread pools or queues are the right
> >> > direction
> >> > > > to go here. They add performance overheads and don't address the
> >> root
> >> > > > causes of the problem.
> >> > > >
> >> > > > There are basically two scenarios where we block, currently. One
> is
> >> > when
> >> > > > we are doing a metadata fetch. I think this is clearly a bug, or
> at
> >> > least
> >> > > > an implementation limitation. From the user's point of view, the
> >> fact
> >> > > that
> >> > > > we are doing a metadata fetch is an implementation detail that
> >> really
> >> > > > shouldn't be exposed like this. We have talked about fixing this
> in
> >> the
> >> > > > past. I think we just should spend the time to do it.
> >> > > >
> >> > > > The second scenario is where the client has produced too much data
> >> in
> >> > too
> >> > > > little time. This could happen if there is a network glitch, or
> the
> >> > > server
> >> > > > is slower than expected. In this case, the behavior is intentional
> >> and
> >> > > not
> >> > > > a bug. To understand this, think abo

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Israel Ekpo]

I would definitely second this though I would like to hear from other
contributors, committers and the PMC as well before any adjustments to the
schedules are finalized.

The KIP-500 work (and all its child KIPs) is crucial to the success of the
next major release (in stability and feature completeness) and the
sentiment is that a lot of users are really looking forward to real
independence from Zookeeper in production so if we need time to do a more
thorough work of reviews, checks, validations and performance testing then
we should proceed to complete that before the 3.0.0 release.

Thanks for the suggestion Ismael and thanks for managing this release,
Konstantine.

The proposed updates to dates makes sense to me.

+1 from me

Sincerely,
Israel




On Wed, May 26, 2021 at 10:19 AM Ismael Juma  wrote:

> Hi Konstantine,
>
> Looking at the schedule and some of the ongoing work in KIP-500 (one of the
> KIPs that is important to land in 3.0), I think we'll need a bit more time.
> We definitely do not want to cause a disruption to our time-based release
> process, but given that this is a major release, I think a small adjustment
> would make sense. I suggest we move the feature freeze and code freeze
> dates by 2 weeks and leave the KIP freeze as it is:
>
> KIP Freeze: 09 Jun 2021
> Feature Freeze: 30 Jun 2021
> Code Freeze: 14 July 2021
>
> This would ensure we have a solid 3.0 release. Thoughts?
>
> Ismael
>
> On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:
>
> > Hi Konstantine,
> >
> > Please Add the following KIPs into Kafka 3.0 plan. In the case of
> KIP-653,
> > it was passed already but not included in the 2.8.0 release.
> >
> > - KIP-653: Upgrade log4j to log4j2
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > - KIP-719: Add Log4J2 Appender
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> >
> > I also updated the proposal documents reflecting the recent changes to
> our
> > codebase. Especially:
> >
> > 1. KIP-653 document
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > now explains which modules will be migrated and why.
> > 2. KIP-719 document
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> > now explains not only the log4j2-appender plan but also upgrading the
> > omitted modules in KIP-653 into log4j2.
> >
> > As you can see here, those two KIPs are the different parts of the same
> > problem. I believe the community will have a good grasp on why both KIPs
> > are best if released altogether.
> >
> > Best,
> > Dongjin
> >
> > On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
> >  wrote:
> >
> > > Hi all,
> > >
> > > Lots of good progress recently towards the 3.0.0 release and given that
> > we
> > > are now a little less than a month away from the first milestone, I
> > wanted
> > > to post a status update and a reminder of the upcoming dates for this
> > major
> > > release.
> > >
> > > According to the current release plan for Apache Kafka 3.0.0:
> > >
> > > KIP Freeze is 09 Jun 2021
> > >
> > > Feature Freeze is 16 Jun 2021
> > >
> > > Code Freeze is 30 Jun 2021
> > >
> > > and at least two weeks of stabilization will follow Code Freeze.
> > >
> > > Since the initial announcement, the list of adopted KIPs targeting
> 3.0.0
> > > has grown with 10 additional KIPs. Several more are under discussion
> and
> > > voting.
> > >
> > > New KIPs will be accepted up until the KIP Freeze and of course the
> > actual
> > > set of features that will make it to 3.0.0 will be finalized right
> after
> > > Feature Freeze.
> > >
> > > With that in mind, if you want to assist the 3.0.0 release process,
> > please
> > > make sure that the adopted KIPs which you aim to include in this major
> > > release have the right number in the “Release” column of the table in
> the
> > > KIP wiki and that their respective Jira tickets include 3.0.0 in the
> "Fix
> > > Version/s" label.
> > >
> > > Kafka Improvement Proposals:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > Release Plan 3.0.0:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > >
> > > Regards,
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > >
> > > > Thank you and hi again.
> > > >
> > > > I just published a release plan at:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > >
> > > > I have included all the currently approved KIPs. I'm expecting this
> > list
> > > > to grow as we approach KIP freeze.
> > > >
> > > > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> > > >
> > > > Please let me know if you have any objections.
> > > >
> > > > Regards,
> > > > Konstantine

Re: [DISCUSS] KIP-655: Windowed "Distinct" Operation for KStream

[Ivan Ponomarev]

Hi John!

I think that your proposal is just fantastic, it simplifies things a lot!

I also felt uncomfortable due to the fact that the proposed `distinct()` 
is not somewhere near `count()` and `reduce(..)`. But 
`selectKey(..).groupByKey().windowedBy(..).distinct()` didn't look like 
a correct option for  me because of the issue with the unneeded 
repartitioning.


The bold idea that we can just CANCEL the repartitioning didn't came to 
my mind.


What seemed to me a single problem is in fact two unrelated problems: 
`distinct` operation and cancelling the unneeded repartitioning.


> what if we introduce a parameter to `selectKey()` that specifies that 
the caller asserts that the new key does _not_ change the data partitioning?


I think a more elegant solution would be not to add a new parameter to 
`selectKey` and all the other key-changing operations (`map`, 
`transform`, `flatMap`, ...), but add a new operator 
`KStream#cancelRepartitioning()` that resets `keyChangingOperation` flag 
for the upstream node. Of course, "use it only if you know what you're 
doing" warning is to be added. Well, it's a topic for a separate KIP!


Concerning `distinct()`. If we use `XXXWindowedKStream` facilities, then 
changes to the API are minimally invasive: we're just adding 
`distinct()` to TimeWindowedKStream and SessionWindowedKStream, and 
that's all.


We can now define `distinct` as an operation that returns only a first 
record that falls into a new window, and filters out all the other 
records that fall into an already existing window. BTW, we can mock the 
behaviour of such an operation with `TopologyTestDriver` using 
`reduce((l, r) -> STOP)`.filterNot((k, v)->STOP.equals(v)).  ;-)


Consider the following example (record times are in seconds):

//three bursts of variously ordered records
4, 5, 6
23, 22, 24
34, 33, 32
//'late arrivals'
7, 22, 35


1. 'Epoch-aligned deduplication' using tumbling windows:

.groupByKey().windowedBy(TimeWindows.of(Duration.ofSeconds(10))).distinct()

produces

(key@[0/1], 4)
(key@[2/3], 23)
(key@[3/4], 34)

-- that is, one record per epoch-aligned window.

2. Hopping and sliding windows do not make much sense here, because they 
produce multiple intersected windows, so that one record can be 
multiplied, but we want deduplication.


3. SessionWindows work for 'data-aligned deduplication'.

.groupByKey().windowedBy(SessionWindows.with(Duration.ofSeconds(10))).distinct() 



produces only

([key@4000/4000], 4)
([key@23000/23000], 23)

because all the records bigger than 7 are stuck together in one session. 
Setting inactivity gap to 9 seconds will return three records:


([key@4000/4000], 4)
([key@23000/23000], 23)
([key@34000/34000], 34)

WDYT? If you like this variant, I will re-write KIP-655 and propose a 
separate KIP for `cancelRepartitioning` (or whatever name we will choose 
for it).


Regards,

Ivan


24.05.2021 22:32, John Roesler пишет:

Hey there, Ivan!

In typical fashion, I'm going to make a somewhat outlandish
proposal. I'm hoping that we can side-step some of the
complications that have arisen. Please bear with me.

It seems like `distinct()` is not fundamentally unlike other windowed
"aggregation" operations. Your concern about unnecessary
repartitioning seems to apply just as well to `count()` as to `distinct()`.
This has come up before, but I don't remember when: what if we
introduce a parameter to `selectKey()` that specifies that the caller
asserts that the new key does _not_ change the data partitioning?
The docs on that parameter would of course spell out all the "rights
and responsibilities" of setting it.

In that case, we could indeed get back to
`selectKey(A).windowBy(B).distinct(...)`, where we get to compose the
key mapper and the windowing function without having to carve out
a separate domain just for `distinct()`. All the rest of the KStream
operations would also benefit.

What do you think?

Thanks,
John

On Sun, May 23, 2021, at 08:09, Ivan Ponomarev wrote:

Hello everyone,

let me revive the discussion for KIP-655. Now I have some time again and
I'm eager to finalize this.

Based on what was already discussed, I think that we can split the
discussion into three topics for our convenience.

The three topics are:

- idExtractor  (how should we extract the deduplication key for the record)

- timeWindows (what time windows should we use)

- miscellaneous (naming etc.)

 idExtractor 

Original proposal: use (k, v) -> f(k, v) mapper, defaulting to (k, v) ->
k.  The drawback here is that we must warn the user to choose such a
function that sets different IDs for records from different partitions,
otherwise same IDs might be not co-partitioned (and not deduplicated as
a result). Additional concern: what should we do when this function
returns null?

Matthias proposed key-only deduplication: that is, no idExtractor at
all, and if we want to use `distinct` for a particular identifier, we
must `selectKey()` before. The dr

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Ismael Juma]

Hi Konstantine,

Looking at the schedule and some of the ongoing work in KIP-500 (one of the
KIPs that is important to land in 3.0), I think we'll need a bit more time.
We definitely do not want to cause a disruption to our time-based release
process, but given that this is a major release, I think a small adjustment
would make sense. I suggest we move the feature freeze and code freeze
dates by 2 weeks and leave the KIP freeze as it is:

KIP Freeze: 09 Jun 2021
Feature Freeze: 30 Jun 2021
Code Freeze: 14 July 2021

This would ensure we have a solid 3.0 release. Thoughts?

Ismael

On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:

> Hi Konstantine,
>
> Please Add the following KIPs into Kafka 3.0 plan. In the case of KIP-653,
> it was passed already but not included in the 2.8.0 release.
>
> - KIP-653: Upgrade log4j to log4j2
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> >
> - KIP-719: Add Log4J2 Appender
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> >
>
> I also updated the proposal documents reflecting the recent changes to our
> codebase. Especially:
>
> 1. KIP-653 document
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> >
> now explains which modules will be migrated and why.
> 2. KIP-719 document
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> >
> now explains not only the log4j2-appender plan but also upgrading the
> omitted modules in KIP-653 into log4j2.
>
> As you can see here, those two KIPs are the different parts of the same
> problem. I believe the community will have a good grasp on why both KIPs
> are best if released altogether.
>
> Best,
> Dongjin
>
> On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
>  wrote:
>
> > Hi all,
> >
> > Lots of good progress recently towards the 3.0.0 release and given that
> we
> > are now a little less than a month away from the first milestone, I
> wanted
> > to post a status update and a reminder of the upcoming dates for this
> major
> > release.
> >
> > According to the current release plan for Apache Kafka 3.0.0:
> >
> > KIP Freeze is 09 Jun 2021
> >
> > Feature Freeze is 16 Jun 2021
> >
> > Code Freeze is 30 Jun 2021
> >
> > and at least two weeks of stabilization will follow Code Freeze.
> >
> > Since the initial announcement, the list of adopted KIPs targeting 3.0.0
> > has grown with 10 additional KIPs. Several more are under discussion and
> > voting.
> >
> > New KIPs will be accepted up until the KIP Freeze and of course the
> actual
> > set of features that will make it to 3.0.0 will be finalized right after
> > Feature Freeze.
> >
> > With that in mind, if you want to assist the 3.0.0 release process,
> please
> > make sure that the adopted KIPs which you aim to include in this major
> > release have the right number in the “Release” column of the table in the
> > KIP wiki and that their respective Jira tickets include 3.0.0 in the "Fix
> > Version/s" label.
> >
> > Kafka Improvement Proposals:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > Release Plan 3.0.0:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> >
> > Regards,
> >
> > Konstantine
> >
> >
> > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > >
> > > Thank you and hi again.
> > >
> > > I just published a release plan at:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > >
> > > I have included all the currently approved KIPs. I'm expecting this
> list
> > > to grow as we approach KIP freeze.
> > >
> > > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> > >
> > > Please let me know if you have any objections.
> > >
> > > Regards,
> > > Konstantine
> > >
> > >
> > > On Wed, Feb 24, 2021 at 9:45 AM Chia-Ping Tsai 
> > > wrote:
> > >
> > >> Thanks for taking over this hard job! +1
> > >>
> > >> On 2021/02/23 08:02:09, Konstantine Karantasis <
> > konstant...@confluent.io>
> > >> wrote:
> > >> > Hi all,
> > >> >
> > >> > Given that we seem to reach an agreement that the feature release
> > after
> > >> the
> > >> > upcoming 2.8.0 will be 3.0.0, I'd like to volunteer to be the
> release
> > >> > manager for the Apache Kafka 3.0.0 release.
> > >> >
> > >> > It's a major release, so I thought it'd be helpful to start
> planning a
> > >> bit
> > >> > in advance.
> > >> >
> > >> > If there are no objections, I'll start working on a release plan in
> > the
> > >> > next few days.
> > >> >
> > >> > Best,
> > >> > Konstantine
> > >> >
> > >>
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinlee

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Josep Prat]

Sorry, I meant `CompletionStage` (instead of CompletableFuture) as this is
the interface.

Best,
———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Wed, May 26, 2021, 15:36 Josep Prat  wrote:

> Hi,
> If I may, I would like to suggest that instead of using Java's `Future`
> class on the API's, it would be better to use `CompletableFuture`. This
> would offer the possibility of applying callbacks on its completion for
> example.
>
> Best,
>
> On Wed, May 26, 2021 at 3:28 PM Matthew de Detrich
>  wrote:
>
>> Maybe there was a miscommunication but I agree with everything you said, I
>> was just clarifying what my definition of blocking is (because I think
>> there was a misunderstanding).
>>
>> And yes you are right, there is a limited amount of threads which is why
>> blocking is a bad thing because having threads sitting around waiting/not
>> doing anything is a waste of resources but ultimately this is also a
>> performance problem because if you don't block you can simply process more
>> IO tasks on a given machine/instance (hence greater performance).
>>
>> In any case, as is clarified the current behavior of send() needs to be
>> fixed. It's returning a Future but since it's internally blocking and
>> using
>> the caller's thread from an API perspective it gives the incorrect
>> impression that it's asynchronous (when it's not).
>>
>> On Wed, May 26, 2021 at 3:15 PM Ryanne Dolan 
>> wrote:
>>
>> > Matthew, it's more than performance tho. In many frameworks the number
>> of
>> > request threads is purposefully constrained, and blocking one means you
>> > have one less to handle requests with. When you're handling a large
>> amount
>> > of requests with a small number of threads, any blocking can lead to
>> thread
>> > exhaustion.
>> >
>> > For this reason, you'll often see send() wrapped in a future or thread
>> > pool. But it's surprising that this would be required, since send()
>> already
>> > returns a future.
>> >
>> > Additionally, even when send() does not actually block, it does a lot of
>> > work on the caller's thread, which is likewise surprising given a
>> future is
>> > returned. The effect is the same: less threads are available to handle
>> > requests, and you risk thread exhaustion.
>> >
>> > I think we may incidentally improve performance if we introduce an
>> internal
>> > thread pool, but the primary motivation here, at least for me, is to fix
>> > the lie the API is telling, not to improve performance.
>> >
>> > Ryanne
>> >
>> >
>> >
>> > On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
>> >  wrote:
>> >
>> > > I think we may need to clarify terminology here, at least to me
>> blocking
>> > > means suspending a current thread to wait for some operation (which is
>> > > wasteful if we are dealing with IO bound tasks). In other words, the
>> > > "blocking" is an implementation detail on how to wait rather than
>> whether
>> > > we need to wait or not, so to me this is more of a performance
>> question.
>> > >
>> > > In the scenario you describe of kafka clients producing too many
>> > messages,
>> > > as you said buffering is what should be done but I wouldn't classify
>> this
>> > > as blocking.
>> > >
>> > > On Mon, May 24, 2021 at 7:54 PM Colin McCabe 
>> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > I agree that we should give users the option of having a fully async
>> > API,
>> > > > but I don't think external thread pools or queues are the right
>> > direction
>> > > > to go here. They add performance overheads and don't address the
>> root
>> > > > causes of the problem.
>> > > >
>> > > > There are basically two scenarios where we block, currently. One is
>> > when
>> > > > we are doing a metadata fetch. I think this is clearly a bug, or at
>> > least
>> > > > an implementation limitation. From the user's point of view, the
>> fact
>> > > that
>> > > > we are doing a metadata fetch is an implementation detail that
>> really
>> > > > shouldn't be exposed like this. We have talked about fixing this in
>> the
>> > > > past. I think we just should spend the time to do it.
>> > > >
>> > > > The second scenario is where the client has produced too much data
>> in
>> > too
>> > > > little time. This could happen if there is a network glitch, or the
>> > > server
>> > > > is slower than expected. In this case, the behavior is intentional
>> and
>> > > not
>> > > > a bug. To understand this, think about what would happen if we
>> didn't
>> > > > block. We would start buffering more and more data in memory, until
>> > > finally
>> > > > the application died with an out of memory error. That would be
>> > > frustrating
>> > > > for users and wouldn't add to the usability of Kafka.
>> > > >
>> > > > We could potentially have an option to handle the out-of-memory
>> > scenario
>> > > > differently by returning an

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Josep Prat]

Hi,
If I may, I would like to suggest that instead of using Java's `Future`
class on the API's, it would be better to use `CompletableFuture`. This
would offer the possibility of applying callbacks on its completion for
example.

Best,

On Wed, May 26, 2021 at 3:28 PM Matthew de Detrich
 wrote:

> Maybe there was a miscommunication but I agree with everything you said, I
> was just clarifying what my definition of blocking is (because I think
> there was a misunderstanding).
>
> And yes you are right, there is a limited amount of threads which is why
> blocking is a bad thing because having threads sitting around waiting/not
> doing anything is a waste of resources but ultimately this is also a
> performance problem because if you don't block you can simply process more
> IO tasks on a given machine/instance (hence greater performance).
>
> In any case, as is clarified the current behavior of send() needs to be
> fixed. It's returning a Future but since it's internally blocking and using
> the caller's thread from an API perspective it gives the incorrect
> impression that it's asynchronous (when it's not).
>
> On Wed, May 26, 2021 at 3:15 PM Ryanne Dolan 
> wrote:
>
> > Matthew, it's more than performance tho. In many frameworks the number of
> > request threads is purposefully constrained, and blocking one means you
> > have one less to handle requests with. When you're handling a large
> amount
> > of requests with a small number of threads, any blocking can lead to
> thread
> > exhaustion.
> >
> > For this reason, you'll often see send() wrapped in a future or thread
> > pool. But it's surprising that this would be required, since send()
> already
> > returns a future.
> >
> > Additionally, even when send() does not actually block, it does a lot of
> > work on the caller's thread, which is likewise surprising given a future
> is
> > returned. The effect is the same: less threads are available to handle
> > requests, and you risk thread exhaustion.
> >
> > I think we may incidentally improve performance if we introduce an
> internal
> > thread pool, but the primary motivation here, at least for me, is to fix
> > the lie the API is telling, not to improve performance.
> >
> > Ryanne
> >
> >
> >
> > On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
> >  wrote:
> >
> > > I think we may need to clarify terminology here, at least to me
> blocking
> > > means suspending a current thread to wait for some operation (which is
> > > wasteful if we are dealing with IO bound tasks). In other words, the
> > > "blocking" is an implementation detail on how to wait rather than
> whether
> > > we need to wait or not, so to me this is more of a performance
> question.
> > >
> > > In the scenario you describe of kafka clients producing too many
> > messages,
> > > as you said buffering is what should be done but I wouldn't classify
> this
> > > as blocking.
> > >
> > > On Mon, May 24, 2021 at 7:54 PM Colin McCabe 
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I agree that we should give users the option of having a fully async
> > API,
> > > > but I don't think external thread pools or queues are the right
> > direction
> > > > to go here. They add performance overheads and don't address the root
> > > > causes of the problem.
> > > >
> > > > There are basically two scenarios where we block, currently. One is
> > when
> > > > we are doing a metadata fetch. I think this is clearly a bug, or at
> > least
> > > > an implementation limitation. From the user's point of view, the fact
> > > that
> > > > we are doing a metadata fetch is an implementation detail that really
> > > > shouldn't be exposed like this. We have talked about fixing this in
> the
> > > > past. I think we just should spend the time to do it.
> > > >
> > > > The second scenario is where the client has produced too much data in
> > too
> > > > little time. This could happen if there is a network glitch, or the
> > > server
> > > > is slower than expected. In this case, the behavior is intentional
> and
> > > not
> > > > a bug. To understand this, think about what would happen if we didn't
> > > > block. We would start buffering more and more data in memory, until
> > > finally
> > > > the application died with an out of memory error. That would be
> > > frustrating
> > > > for users and wouldn't add to the usability of Kafka.
> > > >
> > > > We could potentially have an option to handle the out-of-memory
> > scenario
> > > > differently by returning an error code immediately rather than
> > blocking.
> > > > Applications would have to be rewritten to handle this properly, but
> it
> > > is
> > > > a possibility. I suspect that most of them wouldn't use this, but we
> > > could
> > > > offer it as a possibility for async purists (which might include
> > certain
> > > > frameworks). The big problem the users would have to solve is what to
> > do
> > > > with the record that they were unable to produce due to the buffer
> full
> > > > issue.
> > > >
> > > > bes

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Matthew de Detrich]

Maybe there was a miscommunication but I agree with everything you said, I
was just clarifying what my definition of blocking is (because I think
there was a misunderstanding).

And yes you are right, there is a limited amount of threads which is why
blocking is a bad thing because having threads sitting around waiting/not
doing anything is a waste of resources but ultimately this is also a
performance problem because if you don't block you can simply process more
IO tasks on a given machine/instance (hence greater performance).

In any case, as is clarified the current behavior of send() needs to be
fixed. It's returning a Future but since it's internally blocking and using
the caller's thread from an API perspective it gives the incorrect
impression that it's asynchronous (when it's not).

On Wed, May 26, 2021 at 3:15 PM Ryanne Dolan  wrote:

> Matthew, it's more than performance tho. In many frameworks the number of
> request threads is purposefully constrained, and blocking one means you
> have one less to handle requests with. When you're handling a large amount
> of requests with a small number of threads, any blocking can lead to thread
> exhaustion.
>
> For this reason, you'll often see send() wrapped in a future or thread
> pool. But it's surprising that this would be required, since send() already
> returns a future.
>
> Additionally, even when send() does not actually block, it does a lot of
> work on the caller's thread, which is likewise surprising given a future is
> returned. The effect is the same: less threads are available to handle
> requests, and you risk thread exhaustion.
>
> I think we may incidentally improve performance if we introduce an internal
> thread pool, but the primary motivation here, at least for me, is to fix
> the lie the API is telling, not to improve performance.
>
> Ryanne
>
>
>
> On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
>  wrote:
>
> > I think we may need to clarify terminology here, at least to me blocking
> > means suspending a current thread to wait for some operation (which is
> > wasteful if we are dealing with IO bound tasks). In other words, the
> > "blocking" is an implementation detail on how to wait rather than whether
> > we need to wait or not, so to me this is more of a performance question.
> >
> > In the scenario you describe of kafka clients producing too many
> messages,
> > as you said buffering is what should be done but I wouldn't classify this
> > as blocking.
> >
> > On Mon, May 24, 2021 at 7:54 PM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I agree that we should give users the option of having a fully async
> API,
> > > but I don't think external thread pools or queues are the right
> direction
> > > to go here. They add performance overheads and don't address the root
> > > causes of the problem.
> > >
> > > There are basically two scenarios where we block, currently. One is
> when
> > > we are doing a metadata fetch. I think this is clearly a bug, or at
> least
> > > an implementation limitation. From the user's point of view, the fact
> > that
> > > we are doing a metadata fetch is an implementation detail that really
> > > shouldn't be exposed like this. We have talked about fixing this in the
> > > past. I think we just should spend the time to do it.
> > >
> > > The second scenario is where the client has produced too much data in
> too
> > > little time. This could happen if there is a network glitch, or the
> > server
> > > is slower than expected. In this case, the behavior is intentional and
> > not
> > > a bug. To understand this, think about what would happen if we didn't
> > > block. We would start buffering more and more data in memory, until
> > finally
> > > the application died with an out of memory error. That would be
> > frustrating
> > > for users and wouldn't add to the usability of Kafka.
> > >
> > > We could potentially have an option to handle the out-of-memory
> scenario
> > > differently by returning an error code immediately rather than
> blocking.
> > > Applications would have to be rewritten to handle this properly, but it
> > is
> > > a possibility. I suspect that most of them wouldn't use this, but we
> > could
> > > offer it as a possibility for async purists (which might include
> certain
> > > frameworks). The big problem the users would have to solve is what to
> do
> > > with the record that they were unable to produce due to the buffer full
> > > issue.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Thu, May 20, 2021, at 10:35, Nakamura wrote:
> > > > >
> > > > > My suggestion was just do this in multiple steps/phases, firstly
> > let's
> > > fix
> > > > > the issue of send being misleadingly asynchronous (i.e. internally
> > its
> > > > > blocking) and then later one we can make the various
> > > > > threadpools configurable with a sane default.
> > > >
> > > > I like that approach. I updated the "Which thread should be
> responsible
> > > for
> > > > waiting" part of KIP-739 to add y

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Ryanne Dolan]

Matthew, it's more than performance tho. In many frameworks the number of
request threads is purposefully constrained, and blocking one means you
have one less to handle requests with. When you're handling a large amount
of requests with a small number of threads, any blocking can lead to thread
exhaustion.

For this reason, you'll often see send() wrapped in a future or thread
pool. But it's surprising that this would be required, since send() already
returns a future.

Additionally, even when send() does not actually block, it does a lot of
work on the caller's thread, which is likewise surprising given a future is
returned. The effect is the same: less threads are available to handle
requests, and you risk thread exhaustion.

I think we may incidentally improve performance if we introduce an internal
thread pool, but the primary motivation here, at least for me, is to fix
the lie the API is telling, not to improve performance.

Ryanne



On Wed, May 26, 2021, 6:51 AM Matthew de Detrich
 wrote:

> I think we may need to clarify terminology here, at least to me blocking
> means suspending a current thread to wait for some operation (which is
> wasteful if we are dealing with IO bound tasks). In other words, the
> "blocking" is an implementation detail on how to wait rather than whether
> we need to wait or not, so to me this is more of a performance question.
>
> In the scenario you describe of kafka clients producing too many messages,
> as you said buffering is what should be done but I wouldn't classify this
> as blocking.
>
> On Mon, May 24, 2021 at 7:54 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I agree that we should give users the option of having a fully async API,
> > but I don't think external thread pools or queues are the right direction
> > to go here. They add performance overheads and don't address the root
> > causes of the problem.
> >
> > There are basically two scenarios where we block, currently. One is when
> > we are doing a metadata fetch. I think this is clearly a bug, or at least
> > an implementation limitation. From the user's point of view, the fact
> that
> > we are doing a metadata fetch is an implementation detail that really
> > shouldn't be exposed like this. We have talked about fixing this in the
> > past. I think we just should spend the time to do it.
> >
> > The second scenario is where the client has produced too much data in too
> > little time. This could happen if there is a network glitch, or the
> server
> > is slower than expected. In this case, the behavior is intentional and
> not
> > a bug. To understand this, think about what would happen if we didn't
> > block. We would start buffering more and more data in memory, until
> finally
> > the application died with an out of memory error. That would be
> frustrating
> > for users and wouldn't add to the usability of Kafka.
> >
> > We could potentially have an option to handle the out-of-memory scenario
> > differently by returning an error code immediately rather than blocking.
> > Applications would have to be rewritten to handle this properly, but it
> is
> > a possibility. I suspect that most of them wouldn't use this, but we
> could
> > offer it as a possibility for async purists (which might include certain
> > frameworks). The big problem the users would have to solve is what to do
> > with the record that they were unable to produce due to the buffer full
> > issue.
> >
> > best,
> > Colin
> >
> >
> > On Thu, May 20, 2021, at 10:35, Nakamura wrote:
> > > >
> > > > My suggestion was just do this in multiple steps/phases, firstly
> let's
> > fix
> > > > the issue of send being misleadingly asynchronous (i.e. internally
> its
> > > > blocking) and then later one we can make the various
> > > > threadpools configurable with a sane default.
> > >
> > > I like that approach. I updated the "Which thread should be responsible
> > for
> > > waiting" part of KIP-739 to add your suggestion as my recommended
> > approach,
> > > thank you!  If no one else has major concerns about that approach, I'll
> > > move the alternatives to "rejected alternatives".
> > >
> > > On Thu, May 20, 2021 at 7:26 AM Matthew de Detrich
> > >  wrote:
> > >
> > > > @
> > > >
> > > > Nakamura
> > > > On Wed, May 19, 2021 at 7:35 PM Nakamura  wrote:
> > > >
> > > > > @Ryanne:
> > > > > In my mind's eye I slightly prefer the throwing the "cannot
> enqueue"
> > > > > exception to satisfying the future immediately with the "cannot
> > enqueue"
> > > > > exception?  But I agree, it would be worth doing more research.
> > > > >
> > > > > @Matthew:
> > > > >
> > > > > > 3. Using multiple thread pools is definitely recommended for
> > different
> > > > > > types of tasks, for serialization which is CPU bound you
> definitely
> > > > would
> > > > > > want to use a bounded thread pool that is fixed by the number of
> > CPU's
> > > > > (or
> > > > > > something along those lines).
> > > > > >
> https://gist.github.com/djspiewak/46b543800958cf61af

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Dongjin Lee]

Hi Konstantine,

I greatly appreciate your guidance. Sure, I will notify you again when
KIP-719 is approved.

Regards,
Dongjin

On Wed, May 26, 2021 at 4:08 PM Konstantine Karantasis
 wrote:

> Hi Dongjin,
>
> Thanks for the update.
>
> Added KIP-653 since it has been adopted. I also added "3.0.0 (WIP)" in the
> Release column of the Adopted KIPs table in
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> because the work (according to KAFKA-9366) is currently in progress.
>
> This is the table I'm currently tracking when adding adopted KIPs
> tentatively into the release plan.
>
> Not including KIP-719 to the release plan just yet because it's still under
> discussion. Happy to add it once we conclude voting on it.
>
> Regards,
> Konstantine
>
>
> On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:
>
> > Hi Konstantine,
> >
> > Please Add the following KIPs into Kafka 3.0 plan. In the case of
> KIP-653,
> > it was passed already but not included in the 2.8.0 release.
> >
> > - KIP-653: Upgrade log4j to log4j2
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > - KIP-719: Add Log4J2 Appender
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> >
> > I also updated the proposal documents reflecting the recent changes to
> our
> > codebase. Especially:
> >
> > 1. KIP-653 document
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> > >
> > now explains which modules will be migrated and why.
> > 2. KIP-719 document
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> > >
> > now explains not only the log4j2-appender plan but also upgrading the
> > omitted modules in KIP-653 into log4j2.
> >
> > As you can see here, those two KIPs are the different parts of the same
> > problem. I believe the community will have a good grasp on why both KIPs
> > are best if released altogether.
> >
> > Best,
> > Dongjin
> >
> > On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
> >  wrote:
> >
> > > Hi all,
> > >
> > > Lots of good progress recently towards the 3.0.0 release and given that
> > we
> > > are now a little less than a month away from the first milestone, I
> > wanted
> > > to post a status update and a reminder of the upcoming dates for this
> > major
> > > release.
> > >
> > > According to the current release plan for Apache Kafka 3.0.0:
> > >
> > > KIP Freeze is 09 Jun 2021
> > >
> > > Feature Freeze is 16 Jun 2021
> > >
> > > Code Freeze is 30 Jun 2021
> > >
> > > and at least two weeks of stabilization will follow Code Freeze.
> > >
> > > Since the initial announcement, the list of adopted KIPs targeting
> 3.0.0
> > > has grown with 10 additional KIPs. Several more are under discussion
> and
> > > voting.
> > >
> > > New KIPs will be accepted up until the KIP Freeze and of course the
> > actual
> > > set of features that will make it to 3.0.0 will be finalized right
> after
> > > Feature Freeze.
> > >
> > > With that in mind, if you want to assist the 3.0.0 release process,
> > please
> > > make sure that the adopted KIPs which you aim to include in this major
> > > release have the right number in the “Release” column of the table in
> the
> > > KIP wiki and that their respective Jira tickets include 3.0.0 in the
> "Fix
> > > Version/s" label.
> > >
> > > Kafka Improvement Proposals:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > >
> > > Release Plan 3.0.0:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> > >
> > > Regards,
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > >
> > > > Thank you and hi again.
> > > >
> > > > I just published a release plan at:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > > >
> > > > I have included all the currently approved KIPs. I'm expecting this
> > list
> > > > to grow as we approach KIP freeze.
> > > >
> > > > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> > > >
> > > > Please let me know if you have any objections.
> > > >
> > > > Regards,
> > > > Konstantine
> > > >
> > > >
> > > > On Wed, Feb 24, 2021 at 9:45 AM Chia-Ping Tsai 
> > > > wrote:
> > > >
> > > >> Thanks for taking over this hard job! +1
> > > >>
> > > >> On 2021/02/23 08:02:09, Konstantine Karantasis <
> > > konstant...@confluent.io>
> > > >> wrote:
> > > >> > Hi all,
> > > >> >
> > > >> > Given that we seem to reach an agreement that the feature release
> > > after
> > > >> the
> > > >> > upcoming 2.8.0 will be 3.0.0, I'd like to volunteer to be the
> > release
> > > >> > manager for the Apache Kafka 3.0.0 release.
> > > >> >
> > > >> > It's a major release, so I thought it'd be helpful to start
> > planning a
> > > >> bit

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

CC'd the +1ers of KIP-653 with detailed context:

When I submitted and got the approval of KIP-653: Upgrade log4j to log4j2
,
I thought the log4j2-appender should not be the scope of the work. But it
was wrong.

Since the VerifiableLog4jAppender tool is built upon log4j-appender, log4j
1.x artifact will co-exist with log4j2 artifact in the classpath within
this scheme. Since the log4j 1.x code is not called anymore, I thought it
is not problematic but actually, it was not - when I started to provide a
preview of KIP-653
, some
users reported that sometimes slf4j fails to find the appropriate binding
within the classpath, resulting fail to append the log message.

To resolve this problem, I subtly adjusted the scope of the work; I
excluded Tools and Trogdor from KIP-653 and extended KIP-719 to take care
of them instead, along with providing log4j2-appender. It is why the
current WIP implementations include some classpath logic in the shell
script and *why KIP-653 only can't complete the log4j2 migration*.

I hope you will check this proposal out.

Best,
Dongjin

On Tue, May 25, 2021 at 10:43 PM Dongjin Lee  wrote:

> Bumping up the discussion thread.
>
> Recently, I updated the document of KIP-653: Upgrade log4j to log4j2
> 
>  (accepted)
> and KIP-719: Add Log4J2 Appender
> 
>  (under
> discussion) reflecting the recent changes to our codebase. Especially:
>
> 1. KIP-653 document
> 
>  now
> explains which modules will be migrated and why.
> 2. KIP-719 document
> 
>  now
> explains not only the log4j2-appender plan but also upgrading the omitted
> modules in KIP-653 into log4j2.
>
> As you can see here, those two KIPs are the different parts of the same
> problem. I believe the community will have a good grasp on why both KIPs
> are best if released altogether.
>
> I will open the voting thread now, and please leave a vote if you are
> interested in this issue.
>
> Best,
> Dongjin
>
> On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:
>
>> Hi Kafka dev,
>>
>> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>>
>> All kinds of feedbacks are greatly appreciated!
>>
>> Best,
>> Dongjin
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>>
>>
>> *github:  github.com/dongjinleekr
>> keybase: https://keybase.io/dongjinleekr
>> linkedin: kr.linkedin.com/in/dongjinleekr
>> speakerdeck: speakerdeck.com/dongjin
>> *
>>
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Matthew de Detrich]

I think we may need to clarify terminology here, at least to me blocking
means suspending a current thread to wait for some operation (which is
wasteful if we are dealing with IO bound tasks). In other words, the
"blocking" is an implementation detail on how to wait rather than whether
we need to wait or not, so to me this is more of a performance question.

In the scenario you describe of kafka clients producing too many messages,
as you said buffering is what should be done but I wouldn't classify this
as blocking.

On Mon, May 24, 2021 at 7:54 PM Colin McCabe  wrote:

> Hi all,
>
> I agree that we should give users the option of having a fully async API,
> but I don't think external thread pools or queues are the right direction
> to go here. They add performance overheads and don't address the root
> causes of the problem.
>
> There are basically two scenarios where we block, currently. One is when
> we are doing a metadata fetch. I think this is clearly a bug, or at least
> an implementation limitation. From the user's point of view, the fact that
> we are doing a metadata fetch is an implementation detail that really
> shouldn't be exposed like this. We have talked about fixing this in the
> past. I think we just should spend the time to do it.
>
> The second scenario is where the client has produced too much data in too
> little time. This could happen if there is a network glitch, or the server
> is slower than expected. In this case, the behavior is intentional and not
> a bug. To understand this, think about what would happen if we didn't
> block. We would start buffering more and more data in memory, until finally
> the application died with an out of memory error. That would be frustrating
> for users and wouldn't add to the usability of Kafka.
>
> We could potentially have an option to handle the out-of-memory scenario
> differently by returning an error code immediately rather than blocking.
> Applications would have to be rewritten to handle this properly, but it is
> a possibility. I suspect that most of them wouldn't use this, but we could
> offer it as a possibility for async purists (which might include certain
> frameworks). The big problem the users would have to solve is what to do
> with the record that they were unable to produce due to the buffer full
> issue.
>
> best,
> Colin
>
>
> On Thu, May 20, 2021, at 10:35, Nakamura wrote:
> > >
> > > My suggestion was just do this in multiple steps/phases, firstly let's
> fix
> > > the issue of send being misleadingly asynchronous (i.e. internally its
> > > blocking) and then later one we can make the various
> > > threadpools configurable with a sane default.
> >
> > I like that approach. I updated the "Which thread should be responsible
> for
> > waiting" part of KIP-739 to add your suggestion as my recommended
> approach,
> > thank you!  If no one else has major concerns about that approach, I'll
> > move the alternatives to "rejected alternatives".
> >
> > On Thu, May 20, 2021 at 7:26 AM Matthew de Detrich
> >  wrote:
> >
> > > @
> > >
> > > Nakamura
> > > On Wed, May 19, 2021 at 7:35 PM Nakamura  wrote:
> > >
> > > > @Ryanne:
> > > > In my mind's eye I slightly prefer the throwing the "cannot enqueue"
> > > > exception to satisfying the future immediately with the "cannot
> enqueue"
> > > > exception?  But I agree, it would be worth doing more research.
> > > >
> > > > @Matthew:
> > > >
> > > > > 3. Using multiple thread pools is definitely recommended for
> different
> > > > > types of tasks, for serialization which is CPU bound you definitely
> > > would
> > > > > want to use a bounded thread pool that is fixed by the number of
> CPU's
> > > > (or
> > > > > something along those lines).
> > > > > https://gist.github.com/djspiewak/46b543800958cf61af6efa8e072bfd5c
> is
> > > a
> > > > > very good guide on this topic
> > > > I think this guide is good in general, but I would be hesitant to
> follow
> > > > its guidance re: offloading serialization without benchmarking it.
> My
> > > > understanding is that context-switches have gotten much cheaper, and
> that
> > > > gains from cache locality are small, but they're not nothing.
> Especially
> > > > if the workload has a very small serialization cost, I wouldn't be
> > > shocked
> > > > if it made it slower.  I feel pretty strongly that we should do more
> > > > research here before unconditionally encouraging serialization in a
> > > > threadpool.  If people think it's important to do it here (eg if we
> think
> > > > it would mean another big API change) then we should start thinking
> about
> > > > what benchmarking we can do to gain higher confidence in this kind of
> > > > change.  However, I don't think it would change semantics as
> > > substantially
> > > > as we're proposing here, so I would vote for pushing this to a
> subsequent
> > > > KIP.
> > > >
> > > Of course, its all down to benchmarking, benchmarking and benchmarking.
> > > Ideally speaking you want to use all of the 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #163

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #162

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-12850) Use JDK16 for builds instead of JDK15

[Josep Prat (Jira)]

Josep Prat created KAFKA-12850:
--

 Summary: Use JDK16 for builds instead of JDK15
 Key: KAFKA-12850
 URL: https://issues.apache.org/jira/browse/KAFKA-12850
 Project: Kafka
  Issue Type: Improvement
  Components: build
Reporter: Josep Prat


Given that JDK15 reached EOL in March 2021, it is probably worth migrating the 
Jenkins build pipelines to use JDK16 instead.

Unless there is a compelling reason to stay with JDK15.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 3.0.0 release

[Konstantine Karantasis]

Hi Dongjin,

Thanks for the update.

Added KIP-653 since it has been adopted. I also added "3.0.0 (WIP)" in the
Release column of the Adopted KIPs table in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
because the work (according to KAFKA-9366) is currently in progress.

This is the table I'm currently tracking when adding adopted KIPs
tentatively into the release plan.

Not including KIP-719 to the release plan just yet because it's still under
discussion. Happy to add it once we conclude voting on it.

Regards,
Konstantine


On Tue, May 25, 2021 at 6:40 AM Dongjin Lee  wrote:

> Hi Konstantine,
>
> Please Add the following KIPs into Kafka 3.0 plan. In the case of KIP-653,
> it was passed already but not included in the 2.8.0 release.
>
> - KIP-653: Upgrade log4j to log4j2
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> >
> - KIP-719: Add Log4J2 Appender
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> >
>
> I also updated the proposal documents reflecting the recent changes to our
> codebase. Especially:
>
> 1. KIP-653 document
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-653%3A+Upgrade+log4j+to+log4j2
> >
> now explains which modules will be migrated and why.
> 2. KIP-719 document
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> >
> now explains not only the log4j2-appender plan but also upgrading the
> omitted modules in KIP-653 into log4j2.
>
> As you can see here, those two KIPs are the different parts of the same
> problem. I believe the community will have a good grasp on why both KIPs
> are best if released altogether.
>
> Best,
> Dongjin
>
> On Fri, May 14, 2021 at 8:51 AM Konstantine Karantasis
>  wrote:
>
> > Hi all,
> >
> > Lots of good progress recently towards the 3.0.0 release and given that
> we
> > are now a little less than a month away from the first milestone, I
> wanted
> > to post a status update and a reminder of the upcoming dates for this
> major
> > release.
> >
> > According to the current release plan for Apache Kafka 3.0.0:
> >
> > KIP Freeze is 09 Jun 2021
> >
> > Feature Freeze is 16 Jun 2021
> >
> > Code Freeze is 30 Jun 2021
> >
> > and at least two weeks of stabilization will follow Code Freeze.
> >
> > Since the initial announcement, the list of adopted KIPs targeting 3.0.0
> > has grown with 10 additional KIPs. Several more are under discussion and
> > voting.
> >
> > New KIPs will be accepted up until the KIP Freeze and of course the
> actual
> > set of features that will make it to 3.0.0 will be finalized right after
> > Feature Freeze.
> >
> > With that in mind, if you want to assist the 3.0.0 release process,
> please
> > make sure that the adopted KIPs which you aim to include in this major
> > release have the right number in the “Release” column of the table in the
> > KIP wiki and that their respective Jira tickets include 3.0.0 in the "Fix
> > Version/s" label.
> >
> > Kafka Improvement Proposals:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > Release Plan 3.0.0:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.0.0
> >
> > Regards,
> >
> > Konstantine
> >
> >
> > On Wed, Mar 10, 2021 at 10:28 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > >
> > > Thank you and hi again.
> > >
> > > I just published a release plan at:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177046466
> > >
> > > I have included all the currently approved KIPs. I'm expecting this
> list
> > > to grow as we approach KIP freeze.
> > >
> > > The KIP freeze date for Apache Kafka 3.0 is June 9th, 2021.
> > >
> > > Please let me know if you have any objections.
> > >
> > > Regards,
> > > Konstantine
> > >
> > >
> > > On Wed, Feb 24, 2021 at 9:45 AM Chia-Ping Tsai 
> > > wrote:
> > >
> > >> Thanks for taking over this hard job! +1
> > >>
> > >> On 2021/02/23 08:02:09, Konstantine Karantasis <
> > konstant...@confluent.io>
> > >> wrote:
> > >> > Hi all,
> > >> >
> > >> > Given that we seem to reach an agreement that the feature release
> > after
> > >> the
> > >> > upcoming 2.8.0 will be 3.0.0, I'd like to volunteer to be the
> release
> > >> > manager for the Apache Kafka 3.0.0 release.
> > >> >
> > >> > It's a major release, so I thought it'd be helpful to start
> planning a
> > >> bit
> > >> > in advance.
> > >> >
> > >> > If there are no objections, I'll start working on a release plan in
> > the
> > >> > next few days.
> > >> >
> > >> > Best,
> > >> > Konstantine
> > >> >
> > >>
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
>