Need JIRA permission to self-assign tickets

[Luke Chen]

Hi devs,
I need the JIRA permission to self-assign tickets.
Please help grant me the permission.

JIRA username: showuon

Thank you very much.
Luke

Request permission to "Create KIP" for user: showuon

[Luke Chen]

Hi Devs,
I’d like to create a KIP and need the access to the wiki page. Please help.
My wiki ID: showuon.

Thank you.
Luke

request to be added to the contributor list

[Luke Chen]

Hi Kafka,
I'd like to become contributor for Apache Kafka project. Please help add me
into the contributor list. If there's anything else I need to do, please
let me know.

Thank you very much.

Luke

Re: Want to remove the archive

[Luke Chen]

Hi Satya,
I think Matthias had replied your mail on 5/11. Please check it here:
https://lists.apache.org/thread.html/r6bd3cf8f092accdf59c4b5b878d62faaa1a9de074b31ecf4cb7a0b64%40%3Cdev.kafka.apache.org%3E

Thanks.

On Wed, May 20, 2020 at 9:08 PM Satya Kotni  wrote:

> Any update on this?
>
> Thanks & Regards
> Satya Kotni
>
> On Mon, 11 May 2020 at 10:54, Satya Kotni  wrote:
>
> >
> > Hi,
> >> Please help me in removing this from the  below archive
> >>
> >> https://www.mail-archive.com/dev@kafka.apache.org/msg104541.html
> >>
> >> Best Regards
> >> Satya Kotni
> >>
> >
>

Re: [ANNOUNCE] New committer: Chia-Ping Tsai

[Luke Chen]

Congratulations! Chia-Ping大大!
Well deserved!

Luke

On Tue, Oct 20, 2020 at 2:30 AM Mickael Maison 
wrote:

> Congrats Chia-Ping!
>
> On Mon, Oct 19, 2020 at 8:29 PM Ismael Juma  wrote:
> >
> > Congratulations Chia-Ping!
> >
> > Ismael
> >
> > On Mon, Oct 19, 2020 at 10:25 AM Guozhang Wang 
> wrote:
> >
> > > Hello all,
> > >
> > > I'm happy to announce that Chia-Ping Tsai has accepted his invitation
> to
> > > become an Apache Kafka committer.
> > >
> > > Chia-Ping has been contributing to Kafka since March 2018 and has made
> 74
> > > commits:
> > >
> > > https://github.com/apache/kafka/commits?author=chia7712
> > >
> > > He's also authored several major improvements, participated in the KIP
> > > discussion and PR reviews as well. His major feature development
> includes:
> > >
> > > * KAFKA-9654: Epoch based ReplicaAlterLogDirsThread creation.
> > > * KAFKA-8334: Spiky offsetCommit latency due to lock contention.
> > > * KIP-331: Add default implementation to close() and configure() for
> serde
> > > * KIP-367: Introduce close(Duration) to Producer and AdminClients
> > > * KIP-338: Support to exclude the internal topics in kafka-topics.sh
> > > command
> > >
> > > In addition, Chia-Ping has demonstrated his great diligence fixing test
> > > failures, his impressive engineering attitude and taste in fixing
> tricky
> > > bugs while keeping simple designs.
> > >
> > > Please join me to congratulate Chia-Ping for all the contributions!
> > >
> > >
> > > -- Guozhang
> > >
>

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

[Luke Chen]

Congratulations, Sophie!
You always provide good review comments for my PRs.
Well deserved!

Luke

On Tue, Oct 20, 2020 at 9:23 AM Rankesh Kumar  wrote:

> Many congratulations, Sophie.
>
> Best regards,
> Rankesh
>
>
> > On 20-Oct-2020, at 12:34 AM, Gwen Shapira  wrote:
> >
> > Congratulations, Sophie!
> >
> > On Mon, Oct 19, 2020 at 9:41 AM Matthias J. Sax 
> wrote:
> >>
> >> Hi all,
> >>
> >> I am excited to announce that A. Sophie Blee-Goldman has accepted her
> >> invitation to become an Apache Kafka committer.
> >>
> >> Sophie is actively contributing to Kafka since Feb 2019 and has
> >> accumulated 140 commits. She authored 4 KIPs in the lead
> >>
> >> - KIP-453: Add close() method to RocksDBConfigSetter
> >> - KIP-445: In-memory Session Store
> >> - KIP-428: Add in-memory window store
> >> - KIP-613: Add end-to-end latency metrics to Streams
> >>
> >> and helped to implement two critical KIPs, 429 (incremental rebalancing)
> >> and 441 (smooth auto-scaling; not just implementation but also design).
> >>
> >> In addition, she participates in basically every Kafka Streams related
> >> KIP discussion, reviewed 142 PRs, and is active on the user mailing
> list.
> >>
> >> Thanks for all the contributions, Sophie!
> >>
> >>
> >> Please join me to congratulate her!
> >> -Matthias
> >>
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
>
>

Re: Contribute to Kafka

[Luke Chen]

Hi Monish,
You need to sign up a JIRA account first:
https://issues.apache.org/jira/browse/KAFKA

And then, reply your JIRA id in the email, to let the Kafka commiters know
which JIRA ID to grant permission.

Thanks.
Luke

On Mon, May 25, 2020 at 11:40 PM Monish  wrote:

> Hi,
>
> I would like to contribute to Kafka code base. From
> https://kafka.apache.org/contributing it looks like I need to get the
> permission first to assign a JIRA ticket to myself to work on. Please let
> me know if you need anything from me to get me started. Thank you!
> 
>
> Regards,
> Monish
> monishprab...@gmail.com
>

Re: New Website Layout

[Luke Chen]

Hi Tom, Ben,
PR is ready to address the issue you saw:
https://github.com/apache/kafka-site/pull/292

Thanks.

On Wed, Aug 12, 2020 at 1:09 AM Tom Bentley  wrote:

> Hi Ben,
>
> Thanks for fixing that. Another problem I've just noticed is a couple of
> garbled headings. E.g. scroll down from
> https://kafka.apache.org/documentation.html#design_compactionbasics and
> the
> "What guarantees does log compaction provide?" section is rendering as
>
> $1 class="anchor-heading">$8$9$10
> <https://kafka.apache.org/documentation.html#$4>
>
> with the . Similar thing in
> https://kafka.apache.org/documentation.html#design_quotas. The source HTML
> looks OK to me.
>
> Kind regards,
>
> Tom
>
> On Mon, Aug 10, 2020 at 2:15 PM Ben Stopford  wrote:
>
> > Good spot. Thanks.
> >
> > On Thu, 6 Aug 2020 at 18:59, Ben Weintraub  wrote:
> >
> > > Plus one to Tom's request - the ability to easily generate links to
> > > specific config options is extremely valuable.
> > >
> > > On Thu, Aug 6, 2020 at 10:09 AM Tom Bentley 
> wrote:
> > >
> > > > Hi Ben,
> > > >
> > > > The documentation for the configs (broker, producer etc) used to
> > function
> > > > as links as well as anchors, which made the url fragments more
> > > > discoverable, because you could click on the link and then copy+paste
> > the
> > > > browser URL:
> > > >
> > > > 
> > > >> > > href="#batch.size">batch.size
> > > > 
> > > >
> > > > What seems to have happened with the new layout is the  tags are
> > > empty,
> > > > and no longer enclose the config name,
> > > >
> > > > 
> > > >   
> > > >   batch.size
> > > > 
> > > >
> > > > meaning you can't click on the link to copy and paste the URL. Could
> > the
> > > > old behaviour be restored?
> > > >
> > > > Thanks,
> > > >
> > > > Tom
> > > >
> > > > On Wed, Aug 5, 2020 at 12:43 PM Luke Chen  wrote:
> > > >
> > > > > When entering streams doc, it'll always show:
> > > > > *You're viewing documentation for an older version of Kafka - check
> > out
> > > > our
> > > > > current documentation here.*
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford 
> > wrote:
> > > > >
> > > > > > Thanks for the PR and feedback Michael. Appreciated.
> > > > > >
> > > > > > On Wed, 5 Aug 2020 at 10:49, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thank you, it looks great!
> > > > > > >
> > > > > > > I found a couple of small issues:
> > > > > > > - It's not rendering correctly with http.
> > > > > > > - It's printing "called" to the console. I opened a PR to
> remove
> > > the
> > > > > > > console.log() call:
> > https://github.com/apache/kafka-site/pull/278
> > > > > > >
> > > > > > > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford 
> > > > wrote:
> > > > > > > >
> > > > > > > > The new website layout has gone live as you may have seen.
> > There
> > > > are
> > > > > a
> > > > > > > > couple of rendering issues in the streams developer guide
> that
> > > > we're
> > > > > > > > getting addressed. If anyone spots anything else could they
> > > please
> > > > > > reply
> > > > > > > to
> > > > > > > > this thread.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Ben
> > > > > > > >
> > > > > > > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hey folks
> > > > > > > > >
> > > > > > > > > We've made some updates to the website's look and feel.
> There
> > > is
> > > > a
> > > > > > > staged
> > > > > > > > > version in the link below.
> > > > > > > > >
> > > > > > > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > > > > > > username: kafka
> > > > > > > > > password: streaming
> > > > > > > > >
> > > > > > > > > Comments welcomed.
> > > > > > > > >
> > > > > > > > > Ben
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Ben Stopford
> > > > > >
> > > > > > Lead Technologist, Office of the CTO
> > > > > >
> > > > > > <https://www.confluent.io>
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Ben Stopford
> >
> > Lead Technologist, Office of the CTO
> >
> > <https://www.confluent.io>
> >
>

Re: New Website Layout

[Luke Chen]

When entering streams doc, it'll always show:
*You're viewing documentation for an older version of Kafka - check out our
current documentation here.*



On Wed, Aug 5, 2020 at 6:44 PM Ben Stopford  wrote:

> Thanks for the PR and feedback Michael. Appreciated.
>
> On Wed, 5 Aug 2020 at 10:49, Mickael Maison 
> wrote:
>
> > Thank you, it looks great!
> >
> > I found a couple of small issues:
> > - It's not rendering correctly with http.
> > - It's printing "called" to the console. I opened a PR to remove the
> > console.log() call: https://github.com/apache/kafka-site/pull/278
> >
> > On Wed, Aug 5, 2020 at 9:45 AM Ben Stopford  wrote:
> > >
> > > The new website layout has gone live as you may have seen. There are a
> > > couple of rendering issues in the streams developer guide that we're
> > > getting addressed. If anyone spots anything else could they please
> reply
> > to
> > > this thread.
> > >
> > > Thanks
> > >
> > > Ben
> > >
> > > On Fri, 26 Jun 2020 at 11:48, Ben Stopford  wrote:
> > >
> > > > Hey folks
> > > >
> > > > We've made some updates to the website's look and feel. There is a
> > staged
> > > > version in the link below.
> > > >
> > > > https://ec2-13-57-18-236.us-west-1.compute.amazonaws.com/
> > > > username: kafka
> > > > password: streaming
> > > >
> > > > Comments welcomed.
> > > >
> > > > Ben
> > > >
> > > >
> >
>
>
> --
>
> Ben Stopford
>
> Lead Technologist, Office of the CTO
>
> 
>

Re: First time patch submitter advice

[Luke Chen]

Hi Michael,
The failed unit test has already handled here:
https://issues.apache.org/jira/browse/KAFKA-10155
https://issues.apache.org/jira/browse/KAFKA-10147

So, maybe you can ignore the test errors and mention the issue number in PR.
Thanks.

Luke

On Mon, Jun 15, 2020 at 3:23 PM Michael Carter <
michael.car...@instaclustr.com> wrote:

> Thanks for the response Gwen, that clarifies things for me.
>
> Regarding the unit test (ReassignPartitionsUnitTest.
> testModifyBrokerThrottles),  it appears to fail quite reliably on trunk as
> well (at least on my machine).
> It looks to me like a new override to
> MockAdminClient.describeConfigs(Collection resources)
> (MockAdminClient.java line 369) introduced in commit
> 48b56e533b3ff22ae0e2cf7fcc649e7df19f2b06 changed the behaviour of this
> method that the unit test relied on.
> I’ve just now put a patch into my branch to make that test pass by calling
> a slightly different version of describeConfigs (that avoids the overridden
> behaviour). It’s probably arguable whether that constitutes a fix or not
> though.
>
> Cheers,
> Michael
>
> > On 15 Jun 2020, at 3:41 pm, Gwen Shapira  wrote:
> >
> > Hi,
> >
> > 1. Unfortunately, you need to get a committer to approve running the
> tests.
> > I just gave the green-light on your PR.
> > 2. You can hope that committers will see your PR, but sometimes things
> get
> > lost. If you know someone who is familiar with that area of the code, it
> is
> > a good idea to ping them.
> > 3. We do have some flaky tests. You can see that Jenkins will run 3
> > parallel builds, if some of them pass and the committer confirms that
> > failures are not related to your code, we are ok to merge. Obviously, if
> > you end up tracking them down and fixing, everyone will be very grateful.
> >
> > Hope this helps,
> >
> > Gwen
> >
> > On Sun, Jun 14, 2020 at 5:52 PM Michael Carter <
> > michael.car...@instaclustr.com> wrote:
> >
> >> Hi all,
> >>
> >> I’ve submitted a patch for the first time(
> >> https://github.com/apache/kafka/pull/8844 <
> >> https://github.com/apache/kafka/pull/8844>), and I have a couple of
> >> questions that I’m hoping someone can help me answer.
> >>
> >> I’m a little unclear what happens after that patch has been submitted.
> The
> >> coding guidelines say Jenkins will run tests automatically, but I don’t
> see
> >> any results anywhere. Have I misunderstood what should happen, or do I
> just
> >> not know where to look?
> >> Should I be attempting to find reviewers for the change myself, or is
> that
> >> done independently of the patch submitter?
> >>
> >> Also, in resolving a couple of conflicts that have arisen after the
> patch
> >> was first submitted, I noticed that there are now failing unit tests
> that
> >> have nothing to do with my change. Is there a convention on how to deal
> >> with these? Should it be something that I try to fix on my branch?
> >>
> >> Any thoughts are appreciated.
> >>
> >> Thanks,
> >> Michael
> >
> >
> >
> > --
> > Gwen Shapira
> > Engineering Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
>
>

Re: How does a consumer know the given partition is removed?

[Luke Chen]

Hi Boyuan,
You can create a *ConsumerRebalanceListener* and do something you want when
*onPartitionsRevoked. *
Please check this java doc for more information:
https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html

Thanks.
Luke

On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang  wrote:

> Hi team,
>
> I'm working on a long run application, which uses the Kafka Consumer API to
> poll messages from a given topic and partition. I'm assigning the topic and
> partition manually by using consumer.assign() API and polling messages by
> using consumer.poll().
>
> One common scenario for my application is that certain partitions could be
> removed outside of my application and my application needs to know one
> partition has been removed to stop processing that partition. My question
> is that is there any way to get the removal information when I do
> consumer.assign() or consumer.poll() or any APIs that I can use?
>
> Thanks for your help!
>

Re: Why many "Load Bug xxx" JIRA bug by Tim?

[Luke Chen]

Hi Matthias,
Thanks for your explanation. I understand now.

Thank you.
Luke

On Thu, Jan 7, 2021 at 2:00 AM Matthias J. Sax  wrote:

> This was a spamming attack.
>
> The user was blocked and the corresponding tickets were deleted. (Cf.
> https://issues.apache.org/jira/browse/INFRA-21268)
>
> The "problem" is, that anybody can create an Jira account and create
> tickets. It's in the spirit of open source and the ASF to not lock down
> Jira, to make it easy for people to report issues.
>
> The drawback is, that stuff like this can happen. It's easy to write a
> bot to spam the Jira board...
>
> Because Jira is managed by the ASF infra-team, Kafka committers/PMC
> cannot block users and thus it takes a little longer to react to an
> issue like this, as we need to wait for the infra team to help out.
>
>
> -Matthias
>
>
> On 1/6/21 1:14 AM, M. Manna wrote:
> > I had to register this as spam and block them. I couldn’t disable it from
> > ASF JiRA.
> >
> >  I’m also curious to know how/why such surge occurred.
> >
> > Regards,
> >
> > On Wed, 6 Jan 2021 at 03:45, Luke Chen  wrote:
> >
> >> Hi,
> >> I received a lot of JIRA notification emails today, and they are all
> >> titled: "Load Bug xxx" by Tim.
> >> The bug content doesn't look like a real bug, they are like generated by
> >> automation.
> >> I'm wondering why that could happen?
> >> Do we have any way to delete them all?
> >>
> >> Thanks.
> >> Luke
> >>
> >
>

Re: kafka.apache.org/ top navigation

[Luke Chen]

Hi Tyler,
This is the right place to discuss it, or in JIRA. I saw there's a similar
issue opened recently: KAFKA-10883
, just not in English.
If you have anything updated or PR opened, please associate/update in the
JIRA ticket. (And help update in English would be better)

As the issue you described, I don't have any preference since it didn't
bother me that much :)
I think opening the first subheading links should be the good enough.

Let's see if there are other opinions.

Thanks.
Luke


On Fri, Dec 25, 2020 at 9:51 AM Tyler Knappe 
wrote:

> There doesn't appear to be an easy way to open an issue about the
> kafka.apache.org site specifically, so I figured this would be a good
> place
> to start.  Please point me in another place, if I should open an issue
> elsewhere.
>
> The top navigation for the kafka.apache.org site is broken.  The links for
> 'Get Started' and 'Community' both do not navigate to a new page or open a
> menu.  If the screen size is shrunk, you instead get a hamburger menu that
> does show navigation (both links don't work, still), but at least there is
> some clue as to what navigation is possible.  I'd be happy to open up a PR
> to help fix this, but I'm not entirely certain what the default action
> should be.
>
> It feels like 'Get Started' should open to https://kafka.apache.org/intro
> but the 'Community's first subheading links to https://kafka-summit.org/,
> which would be a very confusing experience, and perhaps
> https://kafka.apache.org/project is a better default.
>
> If the intention was to pop a menu, to open a dropdown for the Get Started
> and Community links, I haven't been able to find it in
> https://github.com/apache/kafka-site/pull/269, but it is a 27k line
> change.
> :)  I think this is the better option, but would incur a bit more work to
> fix since this would require more design work and some JS to handle the
> interaction.
>
> Regardless, please let me know what the intention was and I'd be happy to
> open a PR to fix the issue.
>
> Thanks,
> -Tyler
>

Re: KAFKA-6181 : Yeva Byzek

[Luke Chen]

Hi Prithvi,
You need the JIRA permission to allow you to assign tickets to yourselves.
I cannot help you to assign a JIRA ticket to you if I don't know your jira
ID, either.
Please check How to contribute
 page clearly. (especially
the *Finding
A Project To Work On* section

Thanks.
Luke


On Mon, Nov 16, 2020 at 3:01 PM Prithvi S  wrote:

> Hi,
>
> Please assign me to the jira.
>
> Regards,
> Prithvi
>

Why many "Load Bug xxx" JIRA bug by Tim?

[Luke Chen]

Hi,
I received a lot of JIRA notification emails today, and they are all
titled: "Load Bug xxx" by Tim.
The bug content doesn't look like a real bug, they are like generated by
automation.
I'm wondering why that could happen?
Do we have any way to delete them all?

Thanks.
Luke

Re: How does a consumer know the given partition is removed?

[Luke Chen]

Hi Bruno,
Thanks for the update!

Hi Boyuan,
For listTopics() method, it'll *always* do a remote call, which will have
performance impact for sure.
For partitionsFor() method, it'll *check cache first*, if not found in
cache, then do a remote call to retrieve the topic partition info.

So, I think partitionsFor() should be a better option for you.

Thanks.
Luke


On Fri, Jan 8, 2021 at 2:38 AM Boyuan Zhang  wrote:

> Thanks, folks!
>
> It seems like partitionsFor() and listTopics() is what I want. Do we have
> performance estimates on these 2 API calls, e.g., the time cost of waiting
> for responses? I would invoke these API along a hot path so I want to have
> a general idea on how bad it could be.
>
> Many thanks to your help!
>
> On Thu, Jan 7, 2021 at 1:44 AM Bruno Cadonna  wrote:
>
> > Hi Luke,
> >
> > I am afraid the ConsumerRebalanceListener will not work in this case
> > since Boyuan assigns the partitions manually. The Java docs you linked
> > state
> >
> > If the consumer directly assigns partitions, those partitions will never
> > be reassigned and this callback is not applicable.
> >
> >
> > Hi Boyuan,
> >
> > The consumer has methods partitionsFor() and listTopics(). Probably
> > there is a better way to get the information you want that I am not
> > aware of.
> >
> > Best,
> > Bruno
> >
> > On 07.01.21 05:09, Luke Chen wrote:
> > > Hi Boyuan,
> > > You can create a *ConsumerRebalanceListener* and do something you want
> > when
> > > *onPartitionsRevoked. *
> > > Please check this java doc for more information:
> > >
> >
> https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang 
> wrote:
> > >
> > >> Hi team,
> > >>
> > >> I'm working on a long run application, which uses the Kafka Consumer
> > API to
> > >> poll messages from a given topic and partition. I'm assigning the
> topic
> > and
> > >> partition manually by using consumer.assign() API and polling messages
> > by
> > >> using consumer.poll().
> > >>
> > >> One common scenario for my application is that certain partitions
> could
> > be
> > >> removed outside of my application and my application needs to know one
> > >> partition has been removed to stop processing that partition. My
> > question
> > >> is that is there any way to get the removal information when I do
> > >> consumer.assign() or consumer.poll() or any APIs that I can use?
> > >>
> > >> Thanks for your help!
> > >>
> > >
> >
>

Re: How to build KAFKA 2.7 on z/OS platform

[Luke Chen]

Hi Rabast,
Gradle is just a build automation tool, and we still build the source code
using javac, so your question:
1. Can I use javac and compile the source code, and pack them with jar
manually
--> Yes

2. which sequence is recommended?
--> You might need to refer to the gradle build dependency to know the
sequence, or try to build on another machine with gradle installed, it'll
print out the build sequence.

3. Is a MAKE file available to rebuild the jar libs using make ?
--> No, I don't think so.

I'd suggest you that the easiest way is to build from other machine with
gradle installed,
and put the output jar file onto the z/OS if you really want to run on z/OS
environment.

Thanks.
Luke

On Sat, Jan 23, 2021 at 12:02 AM Rabast, Matthias
 wrote:

> How can I build Kafka from ksource afka-2.7.0-src.tar) on z/OS v2.3
> platform where gradle is NOT available?
>
> Can I use javac and compile the source code, and pack them with jar
> manually ? which sequence is recommended?
>
> Is a MAKE file available to rebuild the jar libs using make ?
>
>
>
> Thanks for any advice.
>

Re: [apache/kafka] KAFKA-10413 Allow even distribution of lost/new tasks when more than one worker

[Luke Chen]

Hi Committers,
please help Ramens add JIRA account: *ramkrish1489* into contribution list.

Thanks.
Luke

On Wed, Feb 3, 2021 at 2:10 PM Ramesh Krishnan 
wrote:

> Hi Luke,
>
> Please find my Jira ID .
>
> ramkrish1489
>
> Thanks
> Ramens
>
> On Wed, 3 Feb 2021 at 11:21 AM, Luke Chen  wrote:
>
> > Hi Ramesh,
> > Do you have the JIRA account? I cannot find your name.
> > You need to have a JIRA account first, and then the committer will grant
> > your access.
> >
> > Thanks.
> > Luke
> >
> > On Wed, Feb 3, 2021 at 1:03 PM Ramesh Krishnan 
> > wrote:
> >
> > > Hi Konstantine,
> > >
> > > Thanks for reviewing the PR [https://github.com/apache/kafka/pull/9319
> ]
> > > and
> > > merging the change.
> > >
> > > Please help assigning this JIRA to me and so that i can update the fix
> > > versions.
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-10413
> > >
> > > Thanks
> > > Ramesh
> > >
> > >
> > > On Mon, Jan 25, 2021 at 8:03 PM Ramesh Krishnan <
> ramesh.154...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Konstantine,
> > > >
> > > > Please re review the below PR , this is a crucial   piece of
> > > functionality
> > > > that is blocking us from using incremental co operative rebalance.
> > > >
> > > > https://github.com/apache/kafka/pull/9319
> > > >
> > > >
> > > > Thanks
> > > > Ramesh
> > > >
> > > > On Fri, Dec 18, 2020 at 12:20 AM Ramesh Krishnan <
> > > ramesh.154...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Team,
> > > >>
> > > >> Can some help re trigger the build for this PR.
> > > >>
> > > >> https://github.com/apache/kafka/pull/9319
> > > >>
> > > >> Thanks
> > > >> Ramesh
> > > >>
> > > >
> > >
> >
>

Re: [apache/kafka] KAFKA-10413 Allow even distribution of lost/new tasks when more than one worker

[Luke Chen]

Hi Ramesh,
Do you have the JIRA account? I cannot find your name.
You need to have a JIRA account first, and then the committer will grant
your access.

Thanks.
Luke

On Wed, Feb 3, 2021 at 1:03 PM Ramesh Krishnan 
wrote:

> Hi Konstantine,
>
> Thanks for reviewing the PR [https://github.com/apache/kafka/pull/9319]
> and
> merging the change.
>
> Please help assigning this JIRA to me and so that i can update the fix
> versions.
>
> https://issues.apache.org/jira/browse/KAFKA-10413
>
> Thanks
> Ramesh
>
>
> On Mon, Jan 25, 2021 at 8:03 PM Ramesh Krishnan 
> wrote:
>
> > Hi Konstantine,
> >
> > Please re review the below PR , this is a crucial   piece of
> functionality
> > that is blocking us from using incremental co operative rebalance.
> >
> > https://github.com/apache/kafka/pull/9319
> >
> >
> > Thanks
> > Ramesh
> >
> > On Fri, Dec 18, 2020 at 12:20 AM Ramesh Krishnan <
> ramesh.154...@gmail.com>
> > wrote:
> >
> >> Hi Team,
> >>
> >> Can some help re trigger the build for this PR.
> >>
> >> https://github.com/apache/kafka/pull/9319
> >>
> >> Thanks
> >> Ramesh
> >>
> >
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Konstantine,
Thanks for your good comments. I've updated the KIP.

Thank you.
Luke


On Wed, Jun 9, 2021 at 2:42 AM Konstantine Karantasis
 wrote:

> Thanks for the KIP Luke.
>
> Looks good overall. Just a few minor suggestions:
>
> 1. How about replacing
> "Note that this change will also propagate to Connect." with -> "Note that
> this change will also automatically be inherited by sink connectors, like
> any other application that uses Kafka consumers, as long as a consumer
> assignor is not explicitly defined in their configuration."
>
> The current sentence is a bit general and it would be good to avoid any
> confusion with Connect's rebalancing protocol for tasks, which already
> supports a single bounce upgrade given that rebalancing protocols in
> Connect have a linear lineage until today.
>
> 2. Let's remove the placeholder text from the Rejected Alternatives and
> simply state that there aren't any. Unless something is worth mentioning in
> that section.
>
> Thanks,
> Konstantine
>
> On Tue, Jun 8, 2021 at 5:20 AM Luke Chen  wrote:
>
> > Hi Ryan,
> > Thanks for the comments. The KAFKA-12896
> > <https://issues.apache.org/jira/browse/KAFKA-12896> is already set as
> > blocker for V3.0, which means, V3.0 won't be released before KAFKA-12896
> > <https://issues.apache.org/jira/browse/KAFKA-12896> is fixed. I think
> this
> > KIP and KAFKA-12896 <https://issues.apache.org/jira/browse/KAFKA-12896>
> > and
> > work in parallel for V3.0 without conflict.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Jun 8, 2021 at 11:30 AM Ryan Leslie (BLOOMBERG/ 919 3RD A) <
> > rles...@bloomberg.net> wrote:
> >
> > > Hey guys,
> > >
> > > Should open bugs concerning cooperative-sticky also be considered
> > blockers
> > > to making it the default? For example, KAFKA-12896 is perhaps still
> being
> > > investigated:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-12896
> > >
> > > Thanks,
> > >
> > > Ryan
> > >
> > > From: dev@kafka.apache.org At: 06/07/21 19:37:45 UTC-4:00To:
> > > dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as
> the
> > > default assignor
> > >
> > > Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully
> > > enable cooperative rebalancing
> > > by default in 3.0 if we have KAFKA-12477 done in time, and if we don't
> > then
> > > there's no harm as it's
> > > not going to change the behavior.
> > >
> > > On Wed, Jun 2, 2021 at 7:28 PM Luke Chen  wrote:
> > >
> > > > Hi Sophie,
> > > > Thanks for the reminder. Yes, I was thinking this KIP doesn't have to
> > be
> > > > put into a major release since it will be fully backward compatible,
> so
> > > no
> > > > need to push it. Currently, if we want to work on this KIP, we need
> > > > KAFKA-12477 and KAFKA-12487. But you're right, we can at least try
> our
> > > best
> > > > to see if we can make it into V3.0 since cooperative rebalancing is a
> > > major
> > > > improvement. I'll kick off a vote later.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Thu, Jun 3, 2021 at 7:08 AM Sophie Blee-Goldman
> > > >  wrote:
> > > >
> > > > > Hey Luke,
> > > > >
> > > > > It's been a while since the last update on this, which is mostly my
> > > fault
> > > > > for picking up
> > > > > other things in the meantime. I'm planning to get back to work
> > > > > on KAFKA-12477 next
> > > > > week but there are minimal changes to the current implementation
> > given
> > > > the
> > > > > proposal
> > > > > I put forth earlier in this KIP discussion, so I think we're good
> to
> > > go.
> > > > >
> > > > > Although this KIP no longer requires a major release since it
> should
> > be
> > > > > fully compatible, I
> > > > > still hope we can get it in to 3.0 since cooperative rebalancing
> is a
> > > > major
> > > > > improvement to
> > > > > the life of a consumer group (and its operator). Can we make sure
> the
> > > KIP
> > > > > reflects the latest
> > > > > and then kick off a vote by next Monday at the late

Re: [VOTE] KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in 3.0)

[Luke Chen]

Hi Ismael,
Thanks for the KIP!
+1 (non-binding)

Thanks.
Luke

On Tue, Jun 8, 2021 at 8:16 AM Israel Ekpo  wrote:

> I support this as well. The motivation and proposed changes makes sense to
> me
>
> +1 (non-binding)
>
> On Mon, Jun 7, 2021 at 6:28 PM Satish Duggana 
> wrote:
>
> > +1 (non-binding)
> >
> > On Tue, 8 Jun 2021 at 04:49, Konstantine Karantasis
> >  wrote:
> >
> > > +1 (binding)
> > >
> > > Konstantine
> > >
> > > On Mon, Jun 7, 2021 at 6:51 AM Josep Prat  >
> > > wrote:
> > >
> > > > Hi Ismael,
> > > >
> > > > Good KIP, it's a +1 (non binding) from my side.
> > > >
> > > > Best,
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Mon, Jun 7, 2021, 15:49 Ismael Juma  wrote:
> > > >
> > > > > Since the goal is to provide ample warning regarding the future
> > removal
> > > > of
> > > > > Scala 2.12 and the change we're proposing for 3.0 is simply a
> > > > documentation
> > > > > update, I am starting the vote:
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/pages/view
> > page.action?pageId=181308218
> > <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218
> >
> > > > >
> > > > > If there are concerns or objections, feel free to point them out in
> > > this
> > > > > thread or the discuss thread.
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in 3.0)

[Luke Chen]

Hi Ismael,
Thanks for the KIP.
+1 (non-binding)

Thanks.
Luke

On Tue, Jun 8, 2021 at 7:18 AM Konstantine Karantasis
 wrote:

> Thanks for the KIP Ismael. I agree that giving an early enough deprecation
> notice for an upcoming change like this one and doing so in a major release
> is best in this case.
>
> +1 (binding)
>
> Konstantine
>
> On Mon, Jun 7, 2021 at 7:30 AM Ryanne Dolan  wrote:
>
> > +1 (non-binding)
> >
> > Ryanne
> >
> > On Mon, Jun 7, 2021, 9:26 AM Satish Duggana 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, 7 Jun 2021 at 19:30, Dongjin Lee  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > As a note: I think the exact removal schedule may be changed.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Mon, Jun 7, 2021, 10:56 PM Josep Prat  >
> > > > wrote:
> > > >
> > > > > Thanks Ismael, it's a +1 (non-binding) from my side,
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Mon, Jun 7, 2021, 15:51 Ismael Juma  wrote:
> > > > >
> > > > > > Since the goal is to provide ample warning regarding the future
> > > removal
> > > > > of
> > > > > > Java 8 support and the change we're proposing for 3.0 is simply a
> > > > > > documentation update, I am starting the vote:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223
> > > > > >
> > > > > > If there are concerns or objections, feel free to point them out
> in
> > > > this
> > > > > > thread or the discuss thread.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi all,
Bump this thread to see if there's anyone wants to vote or have questions.

Thank you.
Luke

Sophie Blee-Goldman  於 2021年6月8日 週二 上午4:40 寫道：

> +1 (binding)
>
> Thanks Luke
>
> On Mon, Jun 7, 2021 at 3:26 AM Luke Chen  wrote:
>
> > Hi Ismael,
> > Thanks for your comments. I updated the KIP for the "Compatibility,
> Upgrade
> > path" section.
> > Simply put, no special upgrade path is necessary.
> >
> > Thank you.
> > Luke
> >
> >
> > On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:
> >
> > > Hi Luke,
> > >
> > > The KIP is a bit unclear with regards to whether two rolling bounces
> are
> > > required or not. What exactly is being proposed?
> > >
> > > Ismael
> > >
> > > On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > > > range" as the default assignor.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> > > >
> > > > This KIP is still waiting for the prerequisite stories to get
> > completed,
> > > > but it should be soon. Hopefully this can be put into V3.0 since
> > > > cooperative rebalancing is a major
> > > > improvement to the life of a consumer group (and its operator).
> > > >
> > > > The discussion thread can be found here:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Ryan,
Thanks for your good comments. I've listed your comments in "Rejected
Alternatives" in KIP.

1. Some cooperative-sticky related defects might not free before V3.0
→ We've marked important defects as blocker for V3.0, ex: KAFKA-12896.
Please raise any important defect if you found any.

2. Cooperative-sticky assignor is also very new for C/C++ users in
librdkafka, so not many in that community have tried incremental
cooperative yet. And bugs are still recently being worked out there too.
→ Thanks for raising this. I checked this library and found currently
only 1 cooperative-sticky related bug open, which is good (10 bugs are
fixed). Anyway, I think the clients can always change the assignor to other
assignors if there are still bugs in the library.

Thank you.
Luke

On Thu, Jun 10, 2021 at 12:40 PM Ryan Leslie  wrote:

> Thanks for the quick replies, Luke and Sophie.
>
> I've not voted, but I agree with accepting the KIP since it's a superior
> feature. I was just reacting mostly to this comment since it didn't mention
> open issues:
>
> > > > Thanks Luke. We may as well get this KIP in to 3.0 so that we can
> fully
> > > > enable cooperative rebalancing
> > > > by default in 3.0 if we have KAFKA-12477 done in time, and if we
> don't
> > > then
> > > > there's no harm as it's
> > > > not going to change the behavior.
>
> But I see now, as Luke said, that the main issue is already considered a
> blocker so it was assumed. Though, I did also wonder if any bugs that may
> have existed since several version ago should actually hold up 3.0, which I
> know is especially about moving away from ZooKeeper.
>
> My sentiment was just that during many release cycles of Kafka since
> cooperative was introduced, there have been issues discovered. And that
> makes sense given that the implementation was complex and quite a lot of
> code changed to make it happen. Hopefully the last of the kinks will have
> been worked out before 3.0. I just wondered if it should be a default in
> 3.0 if it hasn't yet been free of defects for a significant period of time.
> KIP-726 doesn't list any drawbacks for cooperative-sticky, but perhaps this
> is one. I also appreciate that it's already successfully adopted by many,
> particularly streams / connect users. But this may also be where the
> feature has the most benefit due to expensive setup/teardown during
> rebalance, and stop-the-world can be less of a concern for many "regular
> consumers".
>
> This is probably irrelevant here, but another thing to mention is that the
> feature is also very new for C/C++ users in librdkafka, so not many in that
> community have tried incremental cooperative yet. And bugs are still
> recently being worked out there too.
>
> Just playing devil's advocate here, sorry to come across as a negative
> nancy!
>
> On 2021/06/09 00:05:41, Sophie Blee-Goldman 
> wrote:
> > Hey Ryan,
> >
> > Yes, I believe any open bugs regarding the cooperative-sticky assignor
> > should be considered as blockers
> > to it being made the default, if not blockers to the release in general.
> I
> > don't think they need to block the
> > acceptance of this KIP, though, just possibly the implementation of it.
>

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi,
Thanks for the discussion and votes. The KIP passed with:

3  binding votes:
Guozhang Wang
Sophie Blee-Goldman
Konstantine Karantasis

1 non-binding votes:
Dongjin Lee

Thank you.
Luke

On Thu, Jun 10, 2021 at 7:13 AM Konstantine Karantasis
 wrote:

> Thanks for the KIP Luke.
>
> +1 (binding)
>
> Konstantine
>
>
> On Wed, Jun 9, 2021 at 6:36 AM Luke Chen  wrote:
>
> > Hi all,
> > Bump this thread to see if there's anyone wants to vote or have
> questions.
> >
> > Thank you.
> > Luke
> >
> > Sophie Blee-Goldman  於 2021年6月8日 週二 上午4:40
> > 寫道：
> >
> > > +1 (binding)
> > >
> > > Thanks Luke
> > >
> > > On Mon, Jun 7, 2021 at 3:26 AM Luke Chen  wrote:
> > >
> > > > Hi Ismael,
> > > > Thanks for your comments. I updated the KIP for the "Compatibility,
> > > Upgrade
> > > > path" section.
> > > > Simply put, no special upgrade path is necessary.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > >
> > > > On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma 
> wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > The KIP is a bit unclear with regards to whether two rolling
> bounces
> > > are
> > > > > required or not. What exactly is being proposed?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
> > > > >
> > > > > > Hi all,
> > > > > > I'd like to call for a vote on KIP-726: Make the
> > "cooperative-sticky,
> > > > > > range" as the default assignor.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> > > > > >
> > > > > > This KIP is still waiting for the prerequisite stories to get
> > > > completed,
> > > > > > but it should be soon. Hopefully this can be put into V3.0 since
> > > > > > cooperative rebalancing is a major
> > > > > > improvement to the life of a consumer group (and its operator).
> > > > > >
> > > > > > The discussion thread can be found here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

+1 to make it into V3.1.
And the partial and risk-free proposal to make default to ["range",
"cooperative-sticky"] in 3.0 is a brilliant idea.
Sounds good to me.

I can work on it for V3.0 if there's no objections.
Thank you.

Luke

On Wed, Jun 23, 2021 at 12:38 PM Ismael Juma  wrote:

> +1 to making the switch in 3.1 with more time versus rushing it in 3.0.
>
> Ismael
>
> On Tue, Jun 22, 2021 at 8:58 PM Sophie Blee-Goldman
>  wrote:
>
> > I believe we've figured out the root cause of KAFKA-12896
> > <https://issues.apache.org/jira/browse/KAFKA-12896>, and should have a
> fix
> > prepared shortly. See the linked issues for more details.
> >
> > Regarding KIP-726 itself, given that the latest proposal is fully
> > compatible and does not require any breaking changes, we may or may not
> > push to get this into 3.0. Cooperative rebalancing is a huge improvement
> > for the majority of consumer applications, but it's already available for
> > those who want it, so I don't feel too badly about letting it slip from
> 3.0
> > in order to focus on other things. I would also personally feel better
> > about merging it at the beginning of 3.1 so we have the entire release
> > cycle to flush out any potential regressions, rather than rushing it in
> at
> > the last moment (even though the majority of the work has been done for a
> > while).
> >
> > If anyone feels strongly about getting this into 3.0, please let me know,
> > as it's definitely still within reach. Otherwise I'll probably aim for
> 3.1
> > instead.
> >
> > Note (@Luke in particular): we could opt for a partial and risk-free
> > improvement in 3.0, that would make it easier for users to turn on
> > cooperative rebalancing without actually enabling it for them (yet). If
> we
> > change the default config to ["range", "cooperative-sticky"] in 3.0, then
> > the RangeAssignor will remain the default but any application can upgrade
> > to the CooperativeStickyAssignor with just a single rolling bounce (to
> > remove the "range" assignor), rather than the usual two. We can then go
> > ahead and fully make the CooperativeStickyAssignor the default assignor
> in
> > 3.1 by changing the default config to ["cooperative-sticky", "range"] as
> > this KIP has proposed.
> >
> > Thoughts?
> >
> > On Thu, Jun 10, 2021 at 1:46 AM Luke Chen  wrote:
> >
> > > Hi Ryan,
> > > Thanks for your good comments. I've listed your comments in "Rejected
> > > Alternatives" in KIP.
> > >
> > > 1. Some cooperative-sticky related defects might not free before V3.0
> > > → We've marked important defects as blocker for V3.0, ex:
> > KAFKA-12896.
> > > Please raise any important defect if you found any.
> > >
> > > 2. Cooperative-sticky assignor is also very new for C/C++ users in
> > > librdkafka, so not many in that community have tried incremental
> > > cooperative yet. And bugs are still recently being worked out there
> too.
> > > → Thanks for raising this. I checked this library and found
> currently
> > > only 1 cooperative-sticky related bug open, which is good (10 bugs are
> > > fixed). Anyway, I think the clients can always change the assignor to
> > other
> > > assignors if there are still bugs in the library.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Jun 10, 2021 at 12:40 PM Ryan Leslie 
> > > wrote:
> > >
> > > > Thanks for the quick replies, Luke and Sophie.
> > > >
> > > > I've not voted, but I agree with accepting the KIP since it's a
> > superior
> > > > feature. I was just reacting mostly to this comment since it didn't
> > > mention
> > > > open issues:
> > > >
> > > > > > > Thanks Luke. We may as well get this KIP in to 3.0 so that we
> can
> > > > fully
> > > > > > > enable cooperative rebalancing
> > > > > > > by default in 3.0 if we have KAFKA-12477 done in time, and if
> we
> > > > don't
> > > > > > then
> > > > > > > there's no harm as it's
> > > > > > > not going to change the behavior.
> > > >
> > > > But I see now, as Luke said, that the main issue is already
> considered
> > a
> > > > blocker so it was assumed. Though, I did also wonder if any bugs that
> > may
> > > > have existed since several ver

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

[Luke Chen]

Congrats! Konstantine!

Luke

On Tue, Jun 22, 2021 at 3:23 PM Dongjin Lee  wrote:

> Congratulations, Konstantine!
>
> Best,
> Dongjin
>
> On Tue, Jun 22, 2021 at 6:52 AM Konstantine Karantasis
>  wrote:
>
> > Thank you Mickael and everyone for the kind replies. I'm honored and
> > excited to help the project from this role as well.
> >
> > Best,
> > Konstantine
> >
> > On Mon, Jun 21, 2021 at 1:35 PM Bruno Cadonna 
> wrote:
> >
> > > Congrats Konstantine!
> > >
> > > Best,
> > > Bruno
> > >
> > > On 21.06.21 20:10, Randall Hauch wrote:
> > > > Congratulations, Konstantine!
> > > >
> > > > On Mon, Jun 21, 2021 at 12:39 PM Walker Carlson
> > > >  wrote:
> > > >
> > > >> Congratulations!
> > > >>
> > > >> On Mon, Jun 21, 2021 at 12:25 PM Dhruvil Shah
> > >  > > >>>
> > > >> wrote:
> > > >>
> > > >>> Congratulations Konstantine! Well deserved!
> > > >>>
> > > >>> On Mon, Jun 21, 2021 at 10:20 AM Boyang Chen <
> > > reluctanthero...@gmail.com
> > > >>>
> > > >>> wrote:
> > > >>>
> > >  Congratulations Konstantine!
> > > 
> > >  On Mon, Jun 21, 2021 at 10:16 AM Matthias J. Sax <
> mj...@apache.org>
> > > >>> wrote:
> > > 
> > > > Congrats!
> > > >
> > > > On 6/21/21 12:57 PM, Raymond Ng wrote:
> > > >> Congrats Konstantine!
> > > >>
> > > >> /Ray
> > > >>
> > > >> On Mon, Jun 21, 2021 at 9:45 AM Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > wrote:
> > > >>
> > > >>> Congratulations Konstantine!
> > > >>>
> > > >>> On Mon, Jun 21, 2021 at 9:37 AM Tom Bentley <
> tbent...@redhat.com
> > >
> > > > wrote:
> > > >>>
> > >  Congratulations Konstantine!
> > > 
> > >  On Mon, Jun 21, 2021 at 5:33 PM David Jacot
> > > >  > > 
> > >  wrote:
> > > 
> > > > Congrats, Konstantine. Well deserved!
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Mon, Jun 21, 2021 at 6:14 PM Ramesh Krishnan <
> > > >>> ramesh.154...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Congrats Konstantine
> > > >>
> > > >> On Mon, 21 Jun 2021 at 8:58 PM, Mickael Maison <
> > >  mimai...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> It's my pleasure to announce that Konstantine Karantasis is
> > > >> now
> > > >>> a
> > > >>> member of the Kafka PMC.
> > > >>>
> > > >>> Konstantine has been a Kafka committer since Feb 2020. He
> has
> > >  remained
> > > >>> active in the community since becoming a committer.
> > > >>>
> > > >>> Congratulations Konstantine!
> > > >>>
> > > >>> Mickael, on behalf of the Apache Kafka PMC
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>>
> > > >>> --
> > > >>> -- Guozhang
> > > >>>
> > > >>
> > > >
> > > 
> > > >>>
> > > >>
> > > >
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: Kafka upgrading help?

[Luke Chen]

Hi,
I can see in V2.4.0, we upgraded zookeeper from 3.4.14 to 3.5.5. So, I
guess before V2.4.0 is good to go since it's just zookeeper minor version
upgrade.
But you still need to have a complete test for the upgrade.

In your case, I think you can expect the new feature: KRaft mode, to remove
zookeeper. FYI
ref: https://github.com/apache/kafka/blob/trunk/config/kraft/README.md

Thanks.
Luke

On Tue, Jun 15, 2021 at 3:43 PM 邮件帮助中心  wrote:

> Hi:
> The Kafka version of our cluster is 0.10.0.1, and the zookeeper
> version is 3.4.6. Now we want to upgrade Kafka, but we can see that the
> zookeeper of the later Kafka version is higher than 3.4.6. Which Kafka
> version can be upgraded without upgrading the zookeeper version?

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Ryan,
Thanks for the comments. The KAFKA-12896
<https://issues.apache.org/jira/browse/KAFKA-12896> is already set as
blocker for V3.0, which means, V3.0 won't be released before KAFKA-12896
<https://issues.apache.org/jira/browse/KAFKA-12896> is fixed. I think this
KIP and KAFKA-12896 <https://issues.apache.org/jira/browse/KAFKA-12896> and
work in parallel for V3.0 without conflict.

Thank you.
Luke

On Tue, Jun 8, 2021 at 11:30 AM Ryan Leslie (BLOOMBERG/ 919 3RD A) <
rles...@bloomberg.net> wrote:

> Hey guys,
>
> Should open bugs concerning cooperative-sticky also be considered blockers
> to making it the default? For example, KAFKA-12896 is perhaps still being
> investigated:
>
> https://issues.apache.org/jira/browse/KAFKA-12896
>
> Thanks,
>
> Ryan
>
> From: dev@kafka.apache.org At: 06/07/21 19:37:45 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the
> default assignor
>
> Thanks Luke. We may as well get this KIP in to 3.0 so that we can fully
> enable cooperative rebalancing
> by default in 3.0 if we have KAFKA-12477 done in time, and if we don't then
> there's no harm as it's
> not going to change the behavior.
>
> On Wed, Jun 2, 2021 at 7:28 PM Luke Chen  wrote:
>
> > Hi Sophie,
> > Thanks for the reminder. Yes, I was thinking this KIP doesn't have to be
> > put into a major release since it will be fully backward compatible, so
> no
> > need to push it. Currently, if we want to work on this KIP, we need
> > KAFKA-12477 and KAFKA-12487. But you're right, we can at least try our
> best
> > to see if we can make it into V3.0 since cooperative rebalancing is a
> major
> > improvement. I'll kick off a vote later.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Jun 3, 2021 at 7:08 AM Sophie Blee-Goldman
> >  wrote:
> >
> > > Hey Luke,
> > >
> > > It's been a while since the last update on this, which is mostly my
> fault
> > > for picking up
> > > other things in the meantime. I'm planning to get back to work
> > > on KAFKA-12477 next
> > > week but there are minimal changes to the current implementation given
> > the
> > > proposal
> > > I put forth earlier in this KIP discussion, so I think we're good to
> go.
> > >
> > > Although this KIP no longer requires a major release since it should be
> > > fully compatible, I
> > > still hope we can get it in to 3.0 since cooperative rebalancing is a
> > major
> > > improvement to
> > > the life of a consumer group (and its operator). Can we make sure the
> KIP
> > > reflects the latest
> > > and then kick off a vote by next Monday at the latest so we can make
> KIP
> > > freeze?
> > >
> > > Thanks!
> > > Sophie
> > >
> > > On Fri, Apr 16, 2021 at 2:33 PM Guozhang Wang 
> > wrote:
> > >
> > > > 1) From user's perspective, it is always possible that a commit
> within
> > > > onPartitionsRevoked throw in practice (e.g. if the member missed the
> > > > previous rebalance where its assigned partitions are already
> > re-assigned)
> > > > -- and the onPartitionsLost was introduced for that exact reason,
> i.e.
> > it
> > > > is primarily for optimizations, but not for correctness guarantees --
> > on
> > > > the other hand, it would be surprising to users to see the commit
> > returns
> > > > and then later found it not going through. Given that, I'd suggest we
> > > still
> > > > throw the exception right away. Regarding the flag itself though, I
> > agree
> > > > that keeping it set until the next succeeded join group makes sense
> to
> > be
> > > > safer.
> > > >
> > > > 2) That's crystal, thank you for the clarification.
> > > >
> > > > On Wed, Apr 14, 2021 at 6:46 PM Sophie Blee-Goldman
> > > >  wrote:
> > > >
> > > > > 1) Once the short-circuit is triggered, the member will downgrade
> to
> > > the
> > > > > EAGER protocol, but
> > > > > won't necessarily try to rejoin the group right away.
> > > > >
> > > > > In the "happy path", the user has implemented #onPartitionsLost
> > > correctly
> > > > > and will not attempt
> > > > > to commit partitions that are lost. And since these partitions have
> > > > indeed
> > > > > been revoked, the user
> > > > > app

Re: [REVIEW REQUESTED] - Pull Requests for KIP-633 and Addition of Missing Javadocs

[Luke Chen]

Hi Israel,
Thanks for submitting the PR.
I've reviewed those 2 PRs and left some comments there.  I also see Sophie
reviewed 1 of them. See if there are other guys available to review.

But I'd like to suggest you look at the Contributing Code Changes

page first:
*Consider identifying committers or other contributors who have worked on
the code being changed. The easiest is to simply follow GitHub's automatic
suggestions. You can add @username in the PR description to ping them
immediately.*

I don't think sending the review request mail to all dev group is a good
idea*, *especially, your PR just opened less than 1 day.

But still, we are glad to see your contribution to Kafka.
Thank you very much!

Luke

On Fri, May 21, 2021 at 5:59 AM Israel Ekpo  wrote:

> I just submitted my first two pull requests for the project today
>
> [KAFKA-12644] Add Missing Class-Level Javadoc to Exception Classes
> https://github.com/apache/kafka/pull/10741
>
> [KAFKA-8613] Set default grace period to 0
> KIP-633: Drop 24 hour default of grace period in Streams
> https://github.com/apache/kafka/pull/10740
>
> When you have a moment, please take a look and share your feedback and
> thoughts.
>
> Thanks
>

[VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi all,
I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
range" as the default assignor.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248

This KIP is still waiting for the prerequisite stories to get completed,
but it should be soon. Hopefully this can be put into V3.0 since
cooperative rebalancing is a major
improvement to the life of a consumer group (and its operator).

The discussion thread can be found here:
https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E

Thank you.
Luke

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

which seems contradictory? If
> > we
> > > > want to have cooperative behavior be the default, then with the
> > > > default ["cooperative-sticky", "range”] the member would start with
> > > > COOPERATIVE protocol right away.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Mon, Apr 12, 2021 at 5:19 AM Chris Egerton
> > >  > > > >
> > > > wrote:
> > > >
> > > > > Whoops, small correction--meant to say
> > > > > ConsumerRebalanceListener::onPartitionsLost, not
> > > > Consumer::onPartitionsLost
> > > > >
> > > > > On Mon, Apr 12, 2021 at 8:17 AM Chris Egerton  >
> > > > wrote:
> > > > >
> > > > > > Hi Sophie,
> > > > > >
> > > > > > This sounds fantastic. I've made a note on KAFKA-12487 about
> being
> > > sure
> > > > > to
> > > > > > implement Consumer::onPartitionsLost to avoid unnecessary task
> > > failures
> > > > > on
> > > > > > consumer protocol downgrade, but besides that, I don't think
> things
> > > > could
> > > > > > get any smoother for Connect users or developers. The automatic
> > > > protocol
> > > > > > upgrade/downgrade behavior appears safe, intuitive, and
> pain-free.
> > > > > >
> > > > > > Really excited for this development and hoping we can see it come
> > to
> > > > > > fruition in time for the 3.0 release!
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Fri, Apr 9, 2021 at 2:43 PM Sophie Blee-Goldman
> > > > > >  wrote:
> > > > > >
> > > > > >> 1) Yes, all of the above will be part of KAFKA-12477 (not
> KIP-726)
> > > > > >>
> > > > > >> 2) No, KAFKA-12638 would be nice to have but I don't think it's
> > > > > >> appropriate
> > > > > >> to remove
> > > > > >> the default implementation of #onPartitionsLost in 3.0 since we
> > > never
> > > > > gave
> > > > > >> any indication
> > > > > >> yet that we intend to remove it
> > > > > >>
> > > > > >> 3) Yes, this would be similar to when a Consumer drops out of
> the
> > > > group.
> > > > > >> It's always been
> > > > > >> possible for a member to miss a rebalance and have its partition
> > be
> > > > > >> reassigned to another
> > > > > >> member, during which time both members would claim to own said
> > > > > partition.
> > > > > >> But this is safe
> > > > > >> because the member who dropped out is blocked from committing
> > > offsets
> > > > on
> > > > > >> that partition.
> > > > > >>
> > > > > >> On Fri, Apr 9, 2021 at 2:46 AM Luke Chen 
> > wrote:
> > > > > >>
> > > > > >> > Hi Sophie,
> > > > > >> > That sounds great to take care of each case I can think of.
> > > > > >> > Questions:
> > > > > >> > 1. Do you mean the short-Circuit will also be implemented in
> > > > > >> KAFKA-12477?
> > > > > >> > 2. I don't think KAFKA-12638 is the blocker of this KIP-726,
> Am
> > I
> > > > > right?
> > > > > >> > 3. So, does that mean we still have possibility to have
> multiple
> > > > > >> consumer
> > > > > >> > owned the same topic partition? And in this situation, we
> avoid
> > > them
> > > > > >> doing
> > > > > >> > committing, and waiting for next rebalance (should be soon).
> Is
> > my
> > > > > >> > understanding correct?
> > > > > >> >
> > > > > >> > Thank you very much for finding this great solution.
> > > > > >> >
> > > > > >> > Luke
> > > > > >> >
> > > > > >> > On Fri, Apr 9, 2021 at 11:37 AM Sophie Blee-Goldman
&

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Luke Chen]

Hi Dongjin,
Thanks for the KIP.
+1 (non-binding)

Thanks.
Luke

On Sun, Jun 6, 2021 at 5:38 AM Dongjin Lee  wrote:

> Hi all,
>
> I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> ReplicaVerificationTool:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[Luke Chen]

Hi Wenbing,
Thanks for the KIP!
+1 (non-binding)

Thanks.
Luke

On Sun, Jun 6, 2021 at 8:16 AM wenbing shen 
wrote:

> Hi all,
>
> I'd like to start a vote on KIP-749 to add two options (--files and
> --files-separator) to ConsoleProducer.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
>
> Thanks,
>
> Wenbing
>

Re: [VOTE] KIP-726: Make the "cooperative-sticky, range" as the default assignor

[Luke Chen]

Hi Ismael,
Thanks for your comments. I updated the KIP for the "Compatibility, Upgrade
path" section.
Simply put, no special upgrade path is necessary.

Thank you.
Luke


On Mon, Jun 7, 2021 at 4:16 PM Ismael Juma  wrote:

> Hi Luke,
>
> The KIP is a bit unclear with regards to whether two rolling bounces are
> required or not. What exactly is being proposed?
>
> Ismael
>
> On Wed, Jun 2, 2021, 8:16 PM Luke Chen  wrote:
>
> > Hi all,
> > I'd like to call for a vote on KIP-726: Make the "cooperative-sticky,
> > range" as the default assignor.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048248
> >
> > This KIP is still waiting for the prerequisite stories to get completed,
> > but it should be soon. Hopefully this can be put into V3.0 since
> > cooperative rebalancing is a major
> > improvement to the life of a consumer group (and its operator).
> >
> > The discussion thread can be found here:
> >
> >
> https://lists.apache.org/thread.html/ref63417ea84a58c9068ea025b3ad38ca2cc340f5818ac07cd83452b7%40%3Cdev.kafka.apache.org%3E
> >
> > Thank you.
> > Luke
> >
>

Re: [DISCUSS] Apache Kafka 3.0.0 release plan with new updated dates

[Luke Chen]

Hi Konstantine,
We've decided that the KIP-726 will be released in V3.1, not V3.0.
KIP-726: Make the "cooperative-sticky, range" as the default assignor

Could you please remove this KIP from the 3.0 release plan wiki page?

Thank you.
Luke

On Wed, Jun 30, 2021 at 8:23 AM Konstantine Karantasis
 wrote:

> Thanks for the update Colin.
> They are now both in the release plan.
>
> Best,
> Konstantine
>
> On Tue, Jun 29, 2021 at 2:55 PM Colin McCabe  wrote:
>
> > Hi Konstantine,
> >
> > Can you please add two KIPs to the 3.0 release plan wiki page?
> >
> > I'm thinking of:
> > KIP-630: Kafka Raft Snapshots
> > KIP-746: Revise KRaft Metadata Records
> >
> > These are marked as 3.0 on the KIP page but I guess we don't have them on
> > the page yet.
> >
> > Many thanks.
> > Colin
> >
> >
> > On Tue, Jun 22, 2021, at 06:29, Josep Prat wrote:
> > > Hi there,
> > >
> > > As the feature freeze date is approaching, I just wanted to kindly ask
> > for
> > > some reviews on the already submitted PR (
> > > https://github.com/apache/kafka/pull/10840) that implements the
> approved
> > > KIP-744 (https://cwiki.apache.org/confluence/x/XIrOCg). The PR has
> been
> > > ready for review for 2 weeks, and I simply want to make sure there is
> > > enough time to address any possible changes that might be requested.
> > >
> > > Thanks in advance and sorry for any inconvenience caused,
> > > --
> > > Josep
> > > On Mon, Jun 21, 2021 at 11:54 PM Konstantine Karantasis
> > >  wrote:
> > >
> > > > Thanks for the update Bruno.
> > > > I've moved KIP-698 to the list of postponed KIPs in the plan.
> > > >
> > > > Konstantine
> > > >
> > > > On Mon, Jun 21, 2021 at 2:30 AM Bruno Cadonna 
> > wrote:
> > > >
> > > > > Hi Konstantine,
> > > > >
> > > > > The implementation of
> > > > >
> > > > > KIP-698: Add Explicit User Initialization of Broker-side State to
> > Kafka
> > > > > Streams
> > > > >
> > > > > will not be ready for 3.0, so you can remove it from the list.
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On 15.06.21 07:33, Konstantine Karantasis wrote:
> > > > > > Done. Moved it into the table of Adopted KIPs targeting 3.0.0 and
> > to
> > > > the
> > > > > > release plan of course.
> > > > > > Thanks for catching this Israel.
> > > > > >
> > > > > > Best,
> > > > > > Konstantine
> > > > > >
> > > > > > On Mon, Jun 14, 2021 at 7:40 PM Israel Ekpo <
> israele...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Konstantine,
> > > > > >>
> > > > > >> One of mine is missing from this list
> > > > > >>
> > > > > >> KIP-633: Drop 24 hour default of grace period in Streams
> > > > > >> Please could you include it?
> > > > > >>
> > > > > >> Voting has already concluded a long time ago
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Jun 14, 2021 at 6:08 PM Konstantine Karantasis
> > > > > >>  wrote:
> > > > > >>
> > > > > >>> Hi all.
> > > > > >>>
> > > > > >>> KIP Freeze for the next major release of Apache Kafka was
> reached
> > > > last
> > > > > >>> week.
> > > > > >>>
> > > > > >>> As of now, 36 KIPs have concluded their voting process and have
> > been
> > > > > >>> adopted.
> > > > > >>> These KIPs are targeting 3.0 (unless it's noted otherwise in
> the
> > > > > release
> > > > > >>> plan) and their inclusion as new features will be finalized
> right
> > > > after
> > > > > >>> Feature Freeze.
> > > > > >>>
> > > > > >>> At the high level, out of these 36 KIPs, 11 have been
> implemented
> > > > > already
> > > > > >>> and 25 are open or in progress.
> > > > > >>> Here is the full list of adopted KIPs:
> > > > > >>>
> > > > > >>> KIP-751: Drop support for Scala 2.12 in Kafka 4.0 (deprecate in
> > 3.0)
> > > > > >>> KIP-750: Drop support for Java 8 in Kafka 4.0 (deprecate in
> 3.0)
> > > > > >>> KIP-746: Revise KRaft Metadata Records
> > > > > >>> KIP-745: Connect API to restart connector and tasks
> > > > > >>> KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> interface
> > with
> > > > > >>> internal implementation
> > > > > >>> KIP-743: Remove config value 0.10.0-2.4 of Streams built-in
> > metrics
> > > > > >> version
> > > > > >>> config
> > > > > >>> KIP-741: Change default serde to be null
> > > > > >>> KIP-740: Clean up public API in TaskId and fix
> > TaskMetadata#taskId()
> > > > > >>> KIP-738: Removal of Connect's internal converter properties
> > > > > >>> KIP-734: Improve AdminClient.listOffsets to return timestamp
> and
> > > > offset
> > > > > >> for
> > > > > >>> the record with the largest timestamp
> > > > > >>> KIP-733: Change Kafka Streams default replication factor config
> > > > > >>> KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > > >>> KIP-730: Producer ID generation in KRaft mode
> > > > > >>> KIP-726: Make the "cooperative-sticky, range" as the default
> > assignor
> > > > > >>> KIP-725: Streamlining configurations for WindowedSerializer and
> > > > > >>> WindowedDeserializer.
> > > > > >>> KIP-724: Drop support for message formats v0 

Re: [VOTE] KIP-735: Increase default consumer session timeout

[Luke Chen]

Hi Jason,
+1 (non-binding)

Really need this KIP to save poor jenkins flaky tests. :)

Luke

On Thu, Apr 29, 2021 at 4:01 PM David Jacot 
wrote:

> +1 (binding)
>
> Thanks for the KIP.
>
> On Thu, Apr 29, 2021 at 2:27 AM Bill Bejeck  wrote:
>
> > Thanks for the KIP Jason, +1(binding)
> >
> > -Bill
> >
> > On Wed, Apr 28, 2021 at 7:47 PM Guozhang Wang 
> wrote:
> >
> > > +1. Thanks Jason！
> > >
> > > On Wed, Apr 28, 2021 at 12:50 PM Gwen Shapira
>  > >
> > > wrote:
> > >
> > > > I love this improvement.
> > > >
> > > > +1 (binding)
> > > >
> > > > On Wed, Apr 28, 2021 at 10:46 AM Jason Gustafson
> > > > 
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on KIP-735:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout
> > > > > .
> > > > > +1
> > > > > from myself obviously
> > > > >
> > > > > -Jason
> > > > >
> > > >
> > > >
> > > > --
> > > > Gwen Shapira
> > > > Engineering Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter | blog
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [DISCUSS] Apache Kafka 2.6.2 release

[Luke Chen]

Hi Sophie,
I found there is 1 issue that should be cherry-picked into 2.6 and 2.7
branches: KAFKA-12312 .
Simply put, *Scala* *2.13.4* is released at the end of 2020, and we
upgraded to it and fixed some compatible issues on this PR
, more specifically, it's here

.
We only merged this fix on *trunk*(which will be on 2.8), but we didn't
tell users (or we didn't know there'll be compatible issues) not to adopt
the latest *Scala* *2.13.4*.

Therefore, I think we should cherry-pick this fix into 2.6 and 2.7
branches. What do you think?

Thank you.
Luke





On Tue, Feb 9, 2021 at 3:10 AM Sophie Blee-Goldman 
wrote:

> Hey all,
>
> Since all outstanding bugfixes seem to have made their way over to the 2.6
> branch by now, I plan to move ahead with cutting an RC. As always, please
> let me know if you uncover and critical or blocker bugs that affect 2.6
>
> Thanks!
> Sophie
>
> On Thu, Jan 28, 2021 at 9:25 AM John Roesler  wrote:
>
> > Thanks so much for stepping up, Sophie!
> >
> > I'm +1
> >
> > -John
> >
> > On Wed, 2021-01-27 at 17:59 -0500, Bill Bejeck wrote:
> > > Thanks for taking this on Sophie. +1
> > >
> > > Bill
> > >
> > > On Wed, Jan 27, 2021 at 5:59 PM Ismael Juma  wrote:
> > >
> > > > Thanks Sophie! +1
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jan 27, 2021 at 2:45 PM Sophie Blee-Goldman <
> > sop...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to volunteer as release manager for a 2.6.2 release. This
> is
> > > > being
> > > > > accelerated
> > > > > to address a critical regression in Kafka Streams for Windows
> users.
> > > > >
> > > > > You can find the release plan on the wiki:
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.6.2
> > > > >
> > > > > Thanks,
> > > > > Sophie
> > > > >
> > > >
> >
> >
> >
>

Re: [DISCUSS] Apache Kafka 2.6.2 release

[Luke Chen]

Hi Ismael,
Yes, I agree it's like an improvement, not a bug. I don't insist on putting
it into 2.6, just want to bring it to your attention.
In my opinion, this issue will block users who adopt the scala 2.13.4 or
later to use Kafka 2.6.
So if we still have time, we can consider to cherry-pick the fix into 2.6
and 2.7.

What do you think?

Thank you.
Luke

On Wed, Feb 10, 2021 at 3:24 AM Ismael Juma  wrote:

> Can you elaborate why this needs to be in 2.6? Seems like an improvement
> versus a critical bug fix.
>
> Ismael
>
> On Mon, Feb 8, 2021 at 6:39 PM Luke Chen  wrote:
>
> > Hi Sophie,
> > I found there is 1 issue that should be cherry-picked into 2.6 and 2.7
> > branches: KAFKA-12312 <https://issues.apache.org/jira/browse/KAFKA-12312
> >.
> > Simply put, *Scala* *2.13.4* is released at the end of 2020, and we
> > upgraded to it and fixed some compatible issues on this PR
> > <https://github.com/apache/kafka/pull/9643>, more specifically, it's
> here
> > <
> >
> https://github.com/apache/kafka/pull/9643/files#diff-fda3fb44e69a19600913bd951431fb0035996c76325b1c1d84d6f34bec281205R292
> > >
> > .
> > We only merged this fix on *trunk*(which will be on 2.8), but we didn't
> > tell users (or we didn't know there'll be compatible issues) not to adopt
> > the latest *Scala* *2.13.4*.
> >
> > Therefore, I think we should cherry-pick this fix into 2.6 and 2.7
> > branches. What do you think?
> >
> > Thank you.
> > Luke
> >
> >
> >
> >
> >
> > On Tue, Feb 9, 2021 at 3:10 AM Sophie Blee-Goldman 
> > wrote:
> >
> > > Hey all,
> > >
> > > Since all outstanding bugfixes seem to have made their way over to the
> > 2.6
> > > branch by now, I plan to move ahead with cutting an RC. As always,
> please
> > > let me know if you uncover and critical or blocker bugs that affect 2.6
> > >
> > > Thanks!
> > > Sophie
> > >
> > > On Thu, Jan 28, 2021 at 9:25 AM John Roesler 
> > wrote:
> > >
> > > > Thanks so much for stepping up, Sophie!
> > > >
> > > > I'm +1
> > > >
> > > > -John
> > > >
> > > > On Wed, 2021-01-27 at 17:59 -0500, Bill Bejeck wrote:
> > > > > Thanks for taking this on Sophie. +1
> > > > >
> > > > > Bill
> > > > >
> > > > > On Wed, Jan 27, 2021 at 5:59 PM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Thanks Sophie! +1
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Jan 27, 2021 at 2:45 PM Sophie Blee-Goldman <
> > > > sop...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to volunteer as release manager for a 2.6.2 release.
> > This
> > > is
> > > > > > being
> > > > > > > accelerated
> > > > > > > to address a critical regression in Kafka Streams for Windows
> > > users.
> > > > > > >
> > > > > > > You can find the release plan on the wiki:
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.6.2
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sophie
> > > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 2.6.2 release

[Luke Chen]

I just saw the defect KAFKA-12312
<https://issues.apache.org/jira/browse/KAFKA-12312>, so I brought it to
your attention.
Do you think it's not a compatibility issue? If not, I think we don't need
to cherry-pick the fix.

Thanks.
Luke

On Wed, Feb 10, 2021 at 11:16 AM Ismael Juma  wrote:

> It's a perf improvement, there was no regression. I think Luke needs to be
> clearer how this impacts users. Luke, are you referring to cases where
> someone runs the broker in an embedded scenario (eg tests)?
>
> Ismael
>
> On Tue, Feb 9, 2021, 6:50 PM Sophie Blee-Goldman 
> wrote:
>
> > What do you think Ismael? I agreed initially because I saw the commit
> > message says it fixes a performance regression. But admittedly I don't
> have
> > much context on this particular issue
> >
> > If it's low risk then I don't have a strong argument against including
> it.
> > However
> > I aim to cut the rc tomorrow or Thursday, and if it hasn't been
> > cherrypicked by then
> > I won't block the release on it.
> >
> > On Tue, Feb 9, 2021 at 4:53 PM Luke Chen  wrote:
> >
> > > Hi Ismael,
> > > Yes, I agree it's like an improvement, not a bug. I don't insist on
> > putting
> > > it into 2.6, just want to bring it to your attention.
> > > In my opinion, this issue will block users who adopt the scala 2.13.4
> or
> > > later to use Kafka 2.6.
> > > So if we still have time, we can consider to cherry-pick the fix into
> 2.6
> > > and 2.7.
> > >
> > > What do you think?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Feb 10, 2021 at 3:24 AM Ismael Juma  wrote:
> > >
> > > > Can you elaborate why this needs to be in 2.6? Seems like an
> > improvement
> > > > versus a critical bug fix.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Feb 8, 2021 at 6:39 PM Luke Chen  wrote:
> > > >
> > > > > Hi Sophie,
> > > > > I found there is 1 issue that should be cherry-picked into 2.6 and
> > 2.7
> > > > > branches: KAFKA-12312 <
> > > https://issues.apache.org/jira/browse/KAFKA-12312
> > > > >.
> > > > > Simply put, *Scala* *2.13.4* is released at the end of 2020, and we
> > > > > upgraded to it and fixed some compatible issues on this PR
> > > > > <https://github.com/apache/kafka/pull/9643>, more specifically,
> it's
> > > > here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/pull/9643/files#diff-fda3fb44e69a19600913bd951431fb0035996c76325b1c1d84d6f34bec281205R292
> > > > > >
> > > > > .
> > > > > We only merged this fix on *trunk*(which will be on 2.8), but we
> > didn't
> > > > > tell users (or we didn't know there'll be compatible issues) not to
> > > adopt
> > > > > the latest *Scala* *2.13.4*.
> > > > >
> > > > > Therefore, I think we should cherry-pick this fix into 2.6 and 2.7
> > > > > branches. What do you think?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 9, 2021 at 3:10 AM Sophie Blee-Goldman <
> > > sop...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > Since all outstanding bugfixes seem to have made their way over
> to
> > > the
> > > > > 2.6
> > > > > > branch by now, I plan to move ahead with cutting an RC. As
> always,
> > > > please
> > > > > > let me know if you uncover and critical or blocker bugs that
> affect
> > > 2.6
> > > > > >
> > > > > > Thanks!
> > > > > > Sophie
> > > > > >
> > > > > > On Thu, Jan 28, 2021 at 9:25 AM John Roesler <
> vvcep...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks so much for stepping up, Sophie!
> > > > > > >
> > > > > > > I'm +1
> > > > > > >
> > > > > > > -John
> > > > > > >
> > > > > > > On Wed, 2021-01-27 at 17:59 -0500, Bill Bejeck wrote:
> > > > > > > > Thanks for taking this on Sophie. +1
> > > > > > > >
> > > > > > > > Bill
> > > > > > > >
> > > > > > > > On Wed, Jan 27, 2021 at 5:59 PM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Sophie! +1
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Wed, Jan 27, 2021 at 2:45 PM Sophie Blee-Goldman <
> > > > > > > sop...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to volunteer as release manager for a 2.6.2
> > release.
> > > > > This
> > > > > > is
> > > > > > > > > being
> > > > > > > > > > accelerated
> > > > > > > > > > to address a critical regression in Kafka Streams for
> > Windows
> > > > > > users.
> > > > > > > > > >
> > > > > > > > > > You can find the release plan on the wiki:
> > > > > > > > > >
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.6.2
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Sophie
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 2.6.2 release

[Luke Chen]

Hi Ismael & Sophie,
Sorry, after some investigation, I think this might not be a defect. To
work with the project with a specific scala version, user might need to use
the same version as the project used. This issue also happened on other
projects using scala, ex: Spark. ref: https://stackoverflow.com/a/61677956.

So, you can continue to cut the rc.

Thank you very much.
Luke

On Wed, Feb 10, 2021 at 11:19 AM Luke Chen  wrote:

> I just saw the defect KAFKA-12312
> <https://issues.apache.org/jira/browse/KAFKA-12312>, so I brought it to
> your attention.
> Do you think it's not a compatibility issue? If not, I think we don't need
> to cherry-pick the fix.
>
> Thanks.
> Luke
>
> On Wed, Feb 10, 2021 at 11:16 AM Ismael Juma  wrote:
>
>> It's a perf improvement, there was no regression. I think Luke needs to be
>> clearer how this impacts users. Luke, are you referring to cases where
>> someone runs the broker in an embedded scenario (eg tests)?
>>
>> Ismael
>>
>> On Tue, Feb 9, 2021, 6:50 PM Sophie Blee-Goldman 
>> wrote:
>>
>> > What do you think Ismael? I agreed initially because I saw the commit
>> > message says it fixes a performance regression. But admittedly I don't
>> have
>> > much context on this particular issue
>> >
>> > If it's low risk then I don't have a strong argument against including
>> it.
>> > However
>> > I aim to cut the rc tomorrow or Thursday, and if it hasn't been
>> > cherrypicked by then
>> > I won't block the release on it.
>> >
>> > On Tue, Feb 9, 2021 at 4:53 PM Luke Chen  wrote:
>> >
>> > > Hi Ismael,
>> > > Yes, I agree it's like an improvement, not a bug. I don't insist on
>> > putting
>> > > it into 2.6, just want to bring it to your attention.
>> > > In my opinion, this issue will block users who adopt the scala 2.13.4
>> or
>> > > later to use Kafka 2.6.
>> > > So if we still have time, we can consider to cherry-pick the fix into
>> 2.6
>> > > and 2.7.
>> > >
>> > > What do you think?
>> > >
>> > > Thank you.
>> > > Luke
>> > >
>> > > On Wed, Feb 10, 2021 at 3:24 AM Ismael Juma 
>> wrote:
>> > >
>> > > > Can you elaborate why this needs to be in 2.6? Seems like an
>> > improvement
>> > > > versus a critical bug fix.
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Mon, Feb 8, 2021 at 6:39 PM Luke Chen  wrote:
>> > > >
>> > > > > Hi Sophie,
>> > > > > I found there is 1 issue that should be cherry-picked into 2.6 and
>> > 2.7
>> > > > > branches: KAFKA-12312 <
>> > > https://issues.apache.org/jira/browse/KAFKA-12312
>> > > > >.
>> > > > > Simply put, *Scala* *2.13.4* is released at the end of 2020, and
>> we
>> > > > > upgraded to it and fixed some compatible issues on this PR
>> > > > > <https://github.com/apache/kafka/pull/9643>, more specifically,
>> it's
>> > > > here
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/kafka/pull/9643/files#diff-fda3fb44e69a19600913bd951431fb0035996c76325b1c1d84d6f34bec281205R292
>> > > > > >
>> > > > > .
>> > > > > We only merged this fix on *trunk*(which will be on 2.8), but we
>> > didn't
>> > > > > tell users (or we didn't know there'll be compatible issues) not
>> to
>> > > adopt
>> > > > > the latest *Scala* *2.13.4*.
>> > > > >
>> > > > > Therefore, I think we should cherry-pick this fix into 2.6 and 2.7
>> > > > > branches. What do you think?
>> > > > >
>> > > > > Thank you.
>> > > > > Luke
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Feb 9, 2021 at 3:10 AM Sophie Blee-Goldman <
>> > > sop...@confluent.io>
>> > > > > wrote:
>> > > > >
>> > > > > > Hey all,
>> > > > > >
>> > > > > > Since all outstanding bugfixes seem to have made their way over
>> to
>> > > the
>> > > > > 2.6
>> > &

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Sophie & Ismael,
Thank you for your feedback.
No problem, let's pause this KIP and wait for this improvement: KAFKA-12477
<https://issues.apache.org/jira/browse/KAFKA-12477>.

Stay tuned :)

Thank you.
Luke

On Tue, Mar 30, 2021 at 3:14 AM Ismael Juma  wrote:

> Hi Sophie,
>
> I didn't analyze the KIP in detail, but the two suggestions you mentioned
> sound like great improvements.
>
> A bit more context: breaking changes for a widely used product like Kafka
> are costly and hence why we try as hard as we can to avoid them. When it
> comes to the brokers, they are often managed by a central group (or they're
> in the Cloud), so they're a bit easier to manage. Even so, it's still
> possible to upgrade from 0.8.x directly to 2.7 since all protocol versions
> are still supported. When it comes to the basic clients (producer,
> consumer, admin client), they're often embedded in applications so we have
> to be even more conservative.
>
> Ismael
>
> On Mon, Mar 29, 2021 at 10:50 AM Sophie Blee-Goldman
>  wrote:
>
> > Ismael,
> >
> > It seems like given 3.0 is a breaking release, we have to rely on users
> > being aware of this and responsible
> > enough to read the upgrade guide. Otherwise we could never ever make any
> > breaking changes beyond just
> > removing deprecated APIs or other compilation-breaking errors that would
> be
> > immediately visible, no?
> >
> > That said, obviously it's better to have a circuit-breaker that will fail
> > fast in case of a user misconfiguration
> > rather than silently corrupting the consumer group state -- eg for two
> > consumers to overlap in their ownership
> > of the same partition(s). We could definitely implement this, and now
> that
> > I think about it this might solve a
> > related problem in KAFKA-12477
> > <https://issues.apache.org/jira/browse/KAFKA-12477>. We just add a new
> > field to the Assignment in which the group leader
> > indicates whether it's on a recent enough version to understand
> cooperative
> > rebalancing. If an upgraded member
> > joins the group, it'll only be allowed to start following the new
> > rebalancing protocol after receiving the go-ahead
> > from the group leader.
> >
> > If we do go ahead and add this new field in the Assignment then I'm
> pretty
> > confident we can reduce the number
> > of required rolling bounces to just one with KAFKA-12477
> > <https://issues.apache.org/jira/browse/KAFKA-12477>. In that case we
> > should
> > be in much better shape to
> > feel good about changing the default to the CooperativeStickyAssignor.
> How
> > does that sound?
> >
> > To be clear, I'm not proposing we do this as part of KIP-726. Here's my
> > take:
> >
> > Let's pause this KIP while I work on making these two improvements in
> > KAFKA-12477 <https://issues.apache.org/jira/browse/KAFKA-12477>. Once I
> > can
> > confirm the
> > short-circuit and single rolling bounce will be available for 3.0, I'll
> > report back on this thread. Then we can move
> > forward with this KIP again.
> >
> > Thoughts?
> > Sophie
> >
> > On Mon, Mar 29, 2021 at 12:01 AM Luke Chen  wrote:
> >
> > > Hi Ismael,
> > > Thanks for your good question. Answer them below:
> > > *1. Are we saying that every consumer upgraded would have to follow the
> > > complex path described in the KIP? *
> > > --> We suggest that every consumer did these 2 steps of rolling
> upgrade.
> > > And after KAFKA-12477 <
> https://issues.apache.org/jira/browse/KAFKA-12477
> > >
> > > is completed, it can be reduced to 1 rolling upgrade.
> > >
> > > *2. what happens if they don't read the instructions and upgrade as
> they
> > > have in the past?*
> > > --> The reason we want 2 steps of rolling upgrade is that we want to
> > avoid
> > > the situation where leader is on old byte-code and only recognize
> > "eager",
> > > but due to compatibility would still be able to deserialize the new
> > > protocol data from newer versioned members, and hence just go ahead and
> > do
> > > the assignment while new versioned members did not revoke their
> > partitions
> > > before joining the group.
> > >
> > > But I'd say, the new default assignor "CooperativeStickyAssignor" was
> > > already introduced in V2.4.0, and it should be long enough for user to
> > > upgrade to the new byte-code to recognize the "cooperative" protocol

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Ismael,
Thanks for your good question. Answer them below:
*1. Are we saying that every consumer upgraded would have to follow the
complex path described in the KIP? *
--> We suggest that every consumer did these 2 steps of rolling upgrade.
And after KAFKA-12477 <https://issues.apache.org/jira/browse/KAFKA-12477>
is completed, it can be reduced to 1 rolling upgrade.

*2. what happens if they don't read the instructions and upgrade as they
have in the past?*
--> The reason we want 2 steps of rolling upgrade is that we want to avoid
the situation where leader is on old byte-code and only recognize "eager",
but due to compatibility would still be able to deserialize the new
protocol data from newer versioned members, and hence just go ahead and do
the assignment while new versioned members did not revoke their partitions
before joining the group.

But I'd say, the new default assignor "CooperativeStickyAssignor" was
already introduced in V2.4.0, and it should be long enough for user to
upgrade to the new byte-code to recognize the "cooperative" protocol.

What do you think?

Thank you.
Luke

On Mon, Mar 29, 2021 at 12:14 PM Ismael Juma  wrote:

> Thanks for the KIP. Are we saying that every consumer upgraded would have
> to follow the complex path described in the KIP? Also, what happens if they
> don't read the instructions and upgrade as they have in the past?
>
> Ismael
>
> On Fri, Mar 26, 2021, 1:53 AM Luke Chen  wrote:
>
> > Hi everyone,
> > 
> >
> > I'd like to discuss the following proposal to make the
> > CooperativeStickyAssignor as the default assignor.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-726%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor
> >
> > Any comments are welcomed.
> >
> > Thank you.
> > Luke
> >
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Sophie,
Thanks for your good suggestion. I've updated in the KIP-726.

Thank you.
Luke

On Sat, Mar 27, 2021 at 3:24 AM Sophie Blee-Goldman
 wrote:

> Thanks for the KIP! I'm 100% on board with this (obviously :P) and the KIP
> itself looks good to me
> overall. Just one clarification I think you should make:
>
> In the *Public Interfaces* section you say "It won't affect the current
> consumers" -- this is only true
> if those current consumers have explicitly set the
> * partition.assignment.strategy *config on their
> clients. If they've been relying on the default thus far, they will need to
> either follow the safe upgrade
> path as described in KIP-429 or else set the
> *partition.assignment.strategy* config
> to one of the non-
> cooperative assignors during the rolling upgrade if they wish to remain on
> EAGER and/or perform
> the upgrade in just a single rolling bounce.
>
> On that note:
> We have some thoughts for improving the upgrade experience to actually
> reduce the
> safe upgrade path to just a single rolling bounce. This is still in
> progress and we need to
> work out all the kinks so I wouldn't commit to a single rolling bounce
> upgrade as part of
> this KIP, but it's worth noting that by 3.0 we may be in an even better
> position with regards
> to this assignor. The ticket is
> https://issues.apache.org/jira/browse/KAFKA-12477
>
> Cheers,
> Sophie
>
> On Fri, Mar 26, 2021 at 1:53 AM Luke Chen  wrote:
>
> > Hi everyone,
> > 
> >
> > I'd like to discuss the following proposal to make the
> > CooperativeStickyAssignor as the default assignor.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-726%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor
> >
> > Any comments are welcomed.
> >
> > Thank you.
> > Luke
> >
>

Re: New Jenkins job for master and release branches

[Luke Chen]

Thanks Ismael!
I love each JDK+Scala combination is built in parallel!

Thanks.
Luke

On Mon, Apr 5, 2021 at 12:24 PM Gwen Shapira 
wrote:

> W00t! This is super awesome. Thank you so much!!!
>
> On Sun, Apr 4, 2021 at 2:22 PM Ismael Juma  wrote:
>
> > Hi all,
> >
> > As part of KAFKA-12614 <
> https://issues.apache.org/jira/browse/KAFKA-12614
> > >,
> > I have created a Jenkinsfile-based job for trunk, 2.8 and future release
> > branches:
> >
> > https://ci-builds.apache.org/job/Kafka/job/kafka/
> >
> > This has several advantages (many of these are already the case for PR
> > builds):
> >
> >1. The configuration is in source control.
> >2. PR and branch build configuration share most of the logic.
> >3. Unstable (tests failed) and unsuccessful (compile, checkstyle, etc.
> >failed) builds are given different status and color (red vs amber).
> >4. Reporting within a build is improved (each stage is shown as part
> of
> >the build graph)
> >5. Improved parallelism (each JDK+Scala combination is built in
> > parallel)
> >6. Release branches get better JDK version coverage (instead of only
> JDK
> >8 as it used to be)
> >7. Instead of creating a new job for each new release, we can adjust
> the
> >configuration to allow the new release branch.
> >
> > There is currently an open PR <
> https://github.com/apache/kafka/pull/10473>
> > to
> > extend the Jenkinsfile with functionality desired for branch builds. Once
> > that is merged and has been shown to work correctly, I will delete legacy
> > Jenkins jobs like:
> >
> >- https://ci-builds.apache.org/job/Kafka/job/kafka-2.8-jdk8/
> >- https://ci-builds.apache.org/job/Kafka/job/kafka-trunk-jdk11/
> >
> > Let me know if you have questions or comments.
> >
> > Ismael
> >
>
>
> --
> Gwen Shapira
> Engineering Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: Question on kafka connect's incremental rebalance algorithm

[Luke Chen]

Hi Ahmed,
I think this bug KAFKA-12495
 is the issue you
described, which is under code review now.
If not, please open another JIRA ticket to track it.

Thanks.
Luke

On Tue, Apr 6, 2021 at 4:18 AM Thouqueer Ahmed <
thouqueer.ah...@maplelabs.com> wrote:

> Hi,
>   What would happen when new worker joins after the
> synchronization barrier ?
>
> As per code -> performTaskAssignment function of IncrementalAssignor ->
> Boolean canRevoke is false when it is called during the 2nd rebalance.
> Hence revocation is skipped and only assignment is done.
> This would lead to imbalance in #tasks/#connectors.
>
> How is this case handled ?
>
> Thanks,
> Ahmed
>
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Sophie,
Making the default to "cooperative-sticky, range" is a smart idea, to
ensure we can at least fall back to rangeAssignor if consumers are not
following our recommended upgrade path. I updated the KIP accordingly.

Hi Chris,
No problem, I updated the KIP to include the change in Connect.

Thank you very much.

Luke

On Thu, Apr 1, 2021 at 3:24 AM Chris Egerton 
wrote:

> Hi all,
>
> @Sophie - I like the sound of the dual-protocol default. The smooth upgrade
> path it permits sounds fantastic!
>
> @Luke - Do you think we can also include Connect in this KIP? Right now we
> don't set any custom partition assignment strategies for the consumer
> groups we bring up for sink tasks, and if we continue to just use the
> default, the assignment strategy for those consumer groups would change on
> Connect clusters once people upgrade to 3.0. I think this is fine (assuming
> we can take care of https://issues.apache.org/jira/browse/KAFKA-12487
> before then, which I'm fairly optimistic about), but it might be worth a
> sentence or two in the KIP explaining that the change in default will
> intentionally propagate to Connect. And, if we think Connect should be left
> out of this change and stay on the range assignor instead, we should
> probably call that fact out in the KIP as well and state that Connect will
> now override the default partition assignment strategy to be the range
> assignor (assuming the user hasn't specified a value for
> consumer.partition.assignment.strategy in their worker config or for
> consumer.override.partition.assignment.strategy in their connector config).
>
> Cheers,
>
> Chris
>
> On Wed, Mar 31, 2021 at 12:18 AM Sophie Blee-Goldman
>  wrote:
>
> > Ok I'm still fleshing out all the details of KAFKA-12477 but I think we
> can
> > simplify some things a bit, and avoid
> > any kind of "fail-fast" which will require user intervention. In fact I
> > think we can avoid requiring the user to make
> > any changes at all for KIP-726, so we don't have to worry about whether
> > they actually read our documentation:
> >
> > Instead of making ["cooperative-sticky"] the default, we change the
> default
> > to ["cooperative-sticky", "range"].
> > Since "range" is the old default, this is equivalent to the first rolling
> > bounce of the safe upgrade path in KIP-429.
> >
> > Of course this also means that under the current protocol selection
> > mechanism we won't actually upgrade to
> > cooperative rebalancing with the default assignor. But that's where
> > KAFKA-12477 will come in.
> >
> > @Guozhang Wang   I'll get back to you with a
> > concrete proposal and answer your questions, I just want to point out
> > that it's possible to side-step the risk of users shooting themselves in
> > the foot (well, at least in this one specific case,
> > obviously they always find a way)
> >
> > On Tue, Mar 30, 2021 at 10:37 AM Guozhang Wang 
> wrote:
> >
> > > Hi Sophie,
> > >
> > > My question is more related to KAFKA-12477, but since your latest
> replies
> > > are on this thread I figured we can follow-up on the same venue. Just
> so
> > I
> > > understand your latest comments above about the approach:
> > >
> > > * I think, we would need to persist this decision so that the group
> would
> > > never go back to the eager protocol, this bit would be written to the
> > > internal topic's assignment message. Is that correct?
> > > * Maybe you can describe the steps, after the group has decided to move
> > > forward with cooperative protocols, when:
> > > 1) a new member joined the group with the old version, and hence only
> > > recognized eager protocol and executing the eager protocol with its
> first
> > > rebalance, what would happen.
> > > 2) in addition to 1), the new member joined the group with the old
> > version
> > > and only recognized the old subscription format, and was selected as
> the
> > > leader, what would happen.
> > >
> > > Guozhang
> > >
> > >
> > >
> > >
> > > On Mon, Mar 29, 2021 at 10:30 PM Luke Chen  wrote:
> > >
> > > > Hi Sophie & Ismael,
> > > > Thank you for your feedback.
> > > > No problem, let's pause this KIP and wait for this improvement:
> > > KAFKA-12477
> > > > <https://issues.apache.org/jira/browse/KAFKA-12477>.
> > > >
> > > > Stay tuned :)
> > > >
> > > > Thank you.
> > > > Luke

[DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi everyone,


I'd like to discuss the following proposal to make the
CooperativeStickyAssignor as the default assignor.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-726%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor

Any comments are welcomed.

Thank you.
Luke

KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi everyone,


I'd like to discuss the following proposal to make the
CooperativeStickyAssignor as the default assignor.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-726%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor

Any comments are welcomed.

Thank you.
Luke

Re: [DISCUSS] KIP-725: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Sagar,
No problem. I changed my KIP to KIP-726, and also added your KIP into the
KIP list table.

I'll send another discussion mail later for KIP-726.

Thanks
Luke

On Fri, Mar 26, 2021 at 4:41 PM Sagar  wrote:

> Hey Luke,
>
> There's a conflict in the KIP numbers. I have also used 725 for the KiP I
> had written last week;
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177047930
> .
> I had incremented the number while creating this but forgot to add it here.
>
> Do you want me to change the KIP number or would you it?
>
> Thanks!
> Sagar.
>
> On Fri, Mar 26, 2021 at 12:53 PM Luke Chen  wrote:
>
> > Hi everyone,
> >
> > I'd like to discuss the following proposal to make the
> > CooperativeStickyAssignor as the default assignor.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-725%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor
> >
> > Any comments are welcomed.
> >
> > Thank you.
> > Luke
> >
>

[DISCUSS] KIP-725: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi everyone,

I'd like to discuss the following proposal to make the
CooperativeStickyAssignor as the default assignor.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-725%3A+Make+the+CooperativeStickyAssignor+as+the+default+assignor

Any comments are welcomed.

Thank you.
Luke

Re: [ANNOUNCE] New Kafka PMC Member: Chia-Ping Tsai

[Luke Chen]

Congratulations Chia-Ping!
恭喜大大！！

Luke

deng ziming  於 2021年3月13日 週六 上午8:52 寫道：

> Congratulations Chia-Ping!
>
> > On Mar 13, 2021, at 05:39, Sophie Blee-Goldman
>  wrote:
> >
> > Congrats Chia-Ping! Thanks for all your contributions
> >
> > On Fri, Mar 12, 2021 at 12:24 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> >> Congratulations Chia-Ping!
> >>
> >> On Fri, Mar 12, 2021 at 7:54 PM Israel Ekpo 
> wrote:
> >>>
> >>> Congrats @Chia-Ping!
> >>>
> >>> On Fri, Mar 12, 2021 at 2:23 PM Guozhang Wang 
> >> wrote:
> >>>
>  Congratulations Chia-Ping! Really glad to have you on the PMC.
> 
> 
>  Guozhang
> 
>  On Fri, Mar 12, 2021 at 11:14 AM Jun Rao 
> >> wrote:
> 
> > Hi, Everyone,
> >
> > Chia-Ping Tsai has been a Kafka committer since Oct. 15,  2020. He
> >> has
>  been
> > very instrumental to the community since becoming a committer. It's
> >> my
> > pleasure to announce that Chia-Ping  is now a member of Kafka PMC.
> >
> > Congratulations Chia-Ping!
> >
> > Jun
> > on behalf of Apache Kafka PMC
> >
> 
> 
>  --
>  -- Guozhang
> 
> >>
>
>

Re: [ANNOUNCE] New committer: Tom Bentley

[Luke Chen]

Congratulations!

Federico Valeri  於 2021年3月16日 週二 上午4:11 寫道：

> Congrats, Tom!
>
> Well deserved.
>
> On Mon, Mar 15, 2021, 8:09 PM Paolo Patierno  wrote:
>
> > Congratulations Tom!
> >
> > Get Outlook for Android
> >
> > 
> > From: Guozhang Wang 
> > Sent: Monday, March 15, 2021 8:02:44 PM
> > To: dev 
> > Subject: Re: [ANNOUNCE] New committer: Tom Bentley
> >
> > Congratulations Tom!
> >
> > Guozhang
> >
> > On Mon, Mar 15, 2021 at 11:25 AM Bill Bejeck 
> > wrote:
> >
> > > Congratulations, Tom!
> > >
> > > -Bill
> > >
> > > On Mon, Mar 15, 2021 at 2:08 PM Bruno Cadonna
>  > >
> > > wrote:
> > >
> > > > Congrats, Tom!
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On 15.03.21 18:59, Mickael Maison wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Tom Bentley as a committer,
> and
> > > > > we are excited to announce that he accepted!
> > > > >
> > > > > Tom first contributed to Apache Kafka in June 2017 and has been
> > > > > actively contributing since February 2020.
> > > > > He has accumulated 52 commits and worked on a number of KIPs. Here
> > are
> > > > > some of the most significant ones:
> > > > > KIP-183: Change PreferredReplicaLeaderElectionCommand to use
> > > > AdminClient
> > > > > KIP-195: AdminClient.createPartitions
> > > > > KIP-585: Filter and Conditional SMTs
> > > > > KIP-621: Deprecate and replace DescribeLogDirsResult.all() and
> > > > .values()
> > > > > KIP-707: The future of KafkaFuture (still in discussion)
> > > > >
> > > > > In addition, he is very active on the mailing list and has helped
> > > > > review many KIPs.
> > > > >
> > > > > Congratulations Tom and thanks for all the contributions!
> > > > >
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: Kafka logos

[Luke Chen]

The 2nd one is with a subtitle: *Distributed publish-subscribe messaging
system*, which I believe is the old one.
The home page log with: *A Distributed Streaming Platform *should be the
newer one.

Please correct me if I'm wrong.
Thanks.
Luke

On Wed, Mar 3, 2021 at 6:42 AM Justin Mclean  wrote:

> HI,
>
> I notice that the Apache Kafka Logo on the Kafka home page [1] is
> different to what is listed here [2]. I'm creating some training material
> and was wondering what is the correct logo to use?
>
> Thanks,
> Justin
>
> 1. https://kafka.apache.org
> 2. http://apache.org/logos/?#kafka
>

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Luke Chen]

Hi Sophie,
That sounds great to take care of each case I can think of.
Questions:
1. Do you mean the short-Circuit will also be implemented in KAFKA-12477?
2. I don't think KAFKA-12638 is the blocker of this KIP-726, Am I right?
3. So, does that mean we still have possibility to have multiple consumer
owned the same topic partition? And in this situation, we avoid them doing
committing, and waiting for next rebalance (should be soon). Is my
understanding correct?

Thank you very much for finding this great solution.

Luke

On Fri, Apr 9, 2021 at 11:37 AM Sophie Blee-Goldman
 wrote:

> Alright, here's the detailed proposal for KAFKA-12477. This assumes we will
> change the default assignor to ["cooperative-sticky", "range"] in KIP-726.
> It also acknowledges that users may attempt any kind of upgrade without
> reading the docs, and so we need to put in safeguards against data
> corruption rather than assume everyone will follow the safe upgrade path.
>
> With this proposal,
> 1) New applications on 3.0 will enable cooperative rebalancing by default
> 2) Existing applications which don’t set an assignor can safely upgrade to
> 3.0 using a single rolling bounce with no extra steps, and will
> automatically transition to cooperative rebalancing
> 3) Existing applications which do set an assignor that uses EAGER can
> likewise upgrade their applications to COOPERATIVE with a single rolling
> bounce
> 4) Once on 3.0, applications can safely go back and forth between EAGER and
> COOPERATIVE
> 5) Applications can safely downgrade away from 3.0
>
> The high-level idea for dynamic protocol upgrades is that the group will
> leverage the assignor selected by the group coordinator to determine when
> it’s safe to upgrade to COOPERATIVE, and trigger a fail-safe to protect the
> group in case of rare events or user misconfiguration. The group
> coordinator selects the most preferred assignor that’s supported by all
> members of the group, so we know that all members will support COOPERATIVE
> once we receive the “cooperative-sticky” assignor after a rebalance. At
> this point, each member can upgrade their own protocol to COOPERATIVE.
> However, there may be situations in which an EAGER member may join the
> group even after upgrading to COOPERATIVE. For example, during a rolling
> upgrade if the last remaining member on the old bytecode misses a
> rebalance, the other members will be allowed to upgrade to COOPERATIVE. If
> the old member rejoins and is chosen to be the group leader before it’s
> upgraded to 3.0, it won’t be aware that the other members of the group have
> not yet revoked their partitions when computing the assignment.
>
> Short Circuit:
> The risk of mixing the cooperative and eager rebalancing protocols is that
> a partition may be assigned to one member while it has yet to be revoked
> from its previous owner. The danger is that the new owner may begin
> processing and committing offsets for this partition while the previous
> owner is also committing offsets in its #onPartitionsRevoked callback,
> which is invoked at the end of the rebalance in the cooperative protocol.
> This can result in these consumers overwriting each other’s offsets and
> getting a corrupted view of the partition. Note that it’s not possible to
> commit during a rebalance, so we can protect against offset corruption by
> blocking further commits after we detect that the group leader may not
> understand COOPERATIVE, but before we invoke #onPartitionsRevoked. This is
> the “short-circuit” — if we detect that the group is in an unsafe state, we
> invoke #onPartitionsLost instead of #onPartitionsRevoked and explicitly
> prevent offsets from being committed on those revoked partitions.
>
> Consumer procedure:
> Upon startup, the consumer will initially select the highest
> commonly-supported protocol across its configured assignors. With
> ["cooperative-sticky", "range”], the initial protocol will be EAGER when
> the member first joins the group. Following a rebalance, each member will
> check the selected assignor. If the chosen assignor supports COOPERATIVE,
> the member can upgrade their used protocol to COOPERATIVE and no further
> action is required. If the member is already on COOPERATIVE but the
> selected assignor does NOT support it, then we need to trigger the
> short-circuit. In this case we will invoke #onPartitionsLost instead of
> #onPartitionsRevoked, and set a flag to block any attempts at committing
> those partitions which have been revoked. If a commit is attempted, as may
> be the case if the user does not implement #onPartitionsLost (see
> KAFKA-12638), we will throw a CommitFailedException which will be bubbled
> up through poll() after completing the rebalance. The member w

Re: [ANNOUNCE] New Committer: Bruno Cadonna

[Luke Chen]

Congrats Bruno!!

Luke

On Thu, Apr 8, 2021 at 9:18 AM Matthias J. Sax  wrote:

> Congrats Bruno! Very well deserved!
>
>
> -Matthias
>
> On 4/7/21 3:51 PM, Bill Bejeck wrote:
> > Congrats Bruno! Well deserved.
> >
> > Bill
> >
> > On Wed, Apr 7, 2021 at 6:34 PM Guozhang Wang  wrote:
> >
> >> Hello all,
> >>
> >> I'm happy to announce that Bruno Cadonna has accepted his invitation to
> >> become an Apache Kafka committer.
> >>
> >> Bruno has been contributing to Kafka since Jan. 2019 and has made 99
> >> commits and more than 80 PR reviews so far:
> >>
> >> https://github.com/apache/kafka/commits?author=cadonna
> >>
> >> He worked on a few key KIPs on Kafka Streams:
> >>
> >> * KIP-471: Expose RocksDB Metrics in Kafka Streams
> >> * KIP-607: Add Metrics to Kafka Streams to Report Properties of RocksDB
> >> * KIP-662: Throw Exception when Source Topics of a Streams App are
> Deleted
> >>
> >> Besides all the code contributions and reviews, he's also done a handful
> >> for the community: multiple Kafka meetup talks in Berlin and Kafka
> Summit
> >> talks, an introductory class to Kafka at Humboldt-Universität zu Berlin
> >> seminars, and have co-authored a paper on Kafka's stream processing
> >> semantics in this year's SIGMOD conference (
> >> https://en.wikipedia.org/wiki/SIGMOD). Bruno has also been quite
> active on
> >> SO channels and AK mailings.
> >>
> >> Please join me to congratulate Bruno for all the contributions!
> >>
> >> -- Guozhang
> >>
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

[Luke Chen]

Congratulations Bill!

Luke

On Thu, Apr 8, 2021 at 9:17 AM Matthias J. Sax  wrote:

> Hi,
>
> It's my pleasure to announce that Bill Bejeck in now a member of the
> Kafka PMC.
>
> Bill has been a Kafka committer since Feb 2019. He has remained
> active in the community since becoming a committer.
>
>
>
> Congratulations Bill!
>
>  -Matthias, on behalf of Apache Kafka PMC
>

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Luke Chen]

Hi Matthias,
+1 (non-binding)
Thanks for the KIP.

Luke


On Fri, Apr 16, 2021 at 3:32 PM Bruno Cadonna  wrote:

> Thanks Matthias,
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 16.04.21 01:03, Jorge Esteban Quilcate Otoya wrote:
> > +1
> >
> > Thanks Matthias!
> >
> > On Thu, 15 Apr 2021, 20:48 Israel Ekpo,  wrote:
> >
> >> Makes perfect sense to me
> >>
> >> +1 as well.
> >>
> >> Thanks Matthias.
> >>
> >>
> >> On Thu, Apr 15, 2021 at 2:41 PM Guozhang Wang 
> wrote:
> >>
> >>> +1 as well. Thanks!
> >>>
> >>> On Wed, Apr 14, 2021 at 4:30 PM Bill Bejeck  wrote:
> >>>
>  Thanks for the KIP Matthias.
> 
>  +1 (binding)
> 
>  -Bill
> 
>  On Wed, Apr 14, 2021 at 7:06 PM Sophie Blee-Goldman
>   wrote:
> 
> > Thanks Matthias. I'm +1 (binding)
> >
> > -Sophie
> >
> > On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax 
>  wrote:
> >
> >> Hi,
> >>
> >> Because this KIP is rather small, I would like to skip a dedicated
> >> discussion thread and call for a vote right way. If there are any
> >> concerns, we can just discuss on this vote thread:
> >>
> >>
> >>
> >
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-733%3A+change+Kafka+Streams+default+replication+factor+config
> >>
> >> Note, that we actually backed this change via
> >>
> >
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722
> >> already.
> >>
> >> However, I felt it might be worth do make this change more explicit
> >>> as
> >> KIP-464 is rather old now.
> >>
> >> Quote from KIP-464:
> >>
> >>> To exploit this new feature in KafkaStreams, we update the
> >> default
> > value
> >> of Streams configuration parameter `replication.factor` from `1` to
>  `-1`.
> >>
> >>
> >>
> >>
> >> -Matthias
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
>

Re: [VOTE] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Luke Chen]

Hi Sophie,
+1 (non-binding)
Thanks for the KIP!

Luke


Matthias J. Sax  於 2021年4月16日 週五 上午11:21 寫道：

> +1 (binding)
>
> On 4/15/21 12:56 PM, Israel Ekpo wrote:
> > 1+ I agree.
> >
> > I think besides just merging the changes, specific attention should be
> > brought to the KIP in the 3.0 release through blogs and tutorials to make
> > the community more aware of the change in 3.0, upcoming removal of
> > deprecated features in 4.0, it's prerequisites (broker version etc) and
> > benefits
> >
> > Thanks for working on this
> >
> > On Thu, Apr 15, 2021 at 2:43 PM Guozhang Wang 
> wrote:
> >
> >> +1 as well (binding). Thanks Sophie!
> >>
> >> On Thu, Apr 15, 2021 at 7:29 AM Bruno Cadonna 
> wrote:
> >>
> >>> Sophie,
> >>>
> >>> Thank you for the KIP!
> >>>
> >>> +1 (binding)
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On 15.04.21 01:59, Sophie Blee-Goldman wrote:
>  Hey all,
> 
>  I'd like to kick off the vote on KIP-732, to deprecate eos-alpha in
> >> Kafka
>  Streams and migrate away from the "eos-beta" terminology by replacing
> >> it
>  with "eos-v2" to shore up user confidence in this feature.
> 
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> 
>  Please respond on the discussion thread if you have any late-breaking
>  concerns or questions.
> 
>  Thanks!
>  Sophie
> 
> >>>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

[Luke Chen]

Congratulations Randall!

Luke

Bill Bejeck  於 2021年4月17日 週六 上午11:33 寫道：

> Congratulations Randall!
>
> -Bill
>
> On Fri, Apr 16, 2021 at 11:10 PM lobo xu  wrote:
>
> > Congrats Randall
> >
>

Re: Splitting partition may cause message loss for consumers

[Luke Chen]

Hi Okada san,
Yes, I agree the "latest" setting in this situation is not good, and we
should document it.
But I don't think we should change the default auto.offset.reset setting to
the earliest.
The auto.offset.reset setting starts before kafka V1.0, which means, there
are already a lot of users using it and get used to it now.

Thanks.
Luke

On Fri, Feb 19, 2021 at 2:24 PM Haruki Okada  wrote:

> Hi, Kafka.
>
> Recently I noticed that splitting partition may cause message delivery loss
> for consumers with auto.offset.reset=latest.
>
> I described details in https://issues.apache.org/jira/browse/KAFKA-12261 .
>
> Since delivery loss is undesired in most cases, I think this should be
> described in ConsumerConfig.AUTO_OFFSET_RESET_DOC (or somewhere) at least.
>
> What do you think?
>
>
> Thanks,
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

Re: About Kafka 2.7.0 source code compilation error

[Luke Chen]

Yes, and also, you can check the readme in kafka github repo for more
details.
ref: https://github.com/apache/kafka/blob/trunk/README.md

Thanks.
Luke

On Fri, Feb 19, 2021 at 9:51 AM deng ziming 
wrote:

> Hello, please use gradlew, for example `./gradlew jar` `./gradlew idea`,
> or you can use the gradle plugin of IDEA.
> The `@ nowarn` warn seems to be related to different version of scala and
> jdk which you can just ignore.
>
> > On Feb 18, 2021, at 16:38, 韩可  han...@cvicse.com>> wrote:
> >
> > Hello!
> >
> > Recently, we need to build a compilation and development environment for
> Kafka 2.7.0 source code. Now the source code can run successfully. However,
> when we execute "gradle install" or "gradle build", we will report an
> error: @ nowarn annotation does not suppress any warnings. The details are
> as follows:
> >
> >
> >
> > If you delete the @ nowarn annotation according to the location of the
> error, you will still report other errors, so please find out if there is a
> good solution, and hope to get your reply. Thank you!
> >
> > The gradle version used is 6.6
> > The scala version used is 2.13.4
> >
>
>

Re: [VOTE] KIP-766: fetch/findSessions queries with open endpoints for SessionStore/WindowStore

[Luke Chen]

Hi all,
Thanks for the discussion and votes. The KIP passed with:

3  binding votes:
Guozhang Wang
Bill Bejeck
John Roesler

1 non-binding votes:
Luke Chen

I'll close the vote and start the implementation.

Thank you all.
Luke



On Thu, Aug 19, 2021 at 10:47 PM Bill Bejeck  wrote:

> Thanks for the KIP Luke, this is a very needed change.
>
> I'm +1 (binding)
>
> -Bill
>
> On Tue, Aug 17, 2021 at 5:39 PM Guozhang Wang  wrote:
>
> > Thanks Luke! +1 as well.
> >
> > On Tue, Aug 17, 2021 at 11:42 AM John Roesler 
> wrote:
> >
> > > Thanks, Luke!
> > >
> > > I'm sorry I missed your discussion thread. The KIP looks
> > > good to me!
> > >
> > > I'm +1 (binding)
> > >
> > > Thanks,
> > > -John
> > >
> > > On Tue, 2021-08-17 at 16:40 +0800, Luke Chen wrote:
> > > > Hi all,
> > > > I'd like to start to vote for
> > > > *KIP-766: fetch/findSessions queries with open endpoints for
> > > > WindowStore/SessionStore*.
> > > >
> > > > This is a follow-up KIP for KIP-763: Range queries with open
> endpoints
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints
> > > >.
> > > > In KIP-763, we focused on *ReadOnlyKeyValueStore*, in this KIP, we'll
> > > focus
> > > > on *ReadOnlySessionStore* and *ReadOnlyWindowStore, *to have open
> > > endpoints
> > > > queries for SessionStore/WindowStore.
> > > >
> > > > The KIP can be found here:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186876596
> > > >
> > > > Thank you.
> > > > Luke
> > >
> > >
> > >
> >
> > --
> > -- Guozhang
> >
>

Re: Kafka Issue

[Luke Chen]

Sorry, I don't have much knowledge on how event Hubs get all the active
topics.
But at least you can list all available topics in your brokers, or check
details of specific topic by these commands.

> bin/kafka-topics.sh --list --bootstrap-server localhost:9092

> bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 
> localhost:9092
Topic:quickstart-events  PartitionCount:1ReplicationFactor:1 Configs:
Topic: quickstart-events Partition: 0Leader: 0   Replicas: 0 Isr: 0

Thanks

Luke


On Fri, Aug 20, 2021 at 3:44 PM Roel Angeles 
wrote:

> Hi Adam and Luke,
>
>
>
> Sending the drive for the screenshots.
>
>
> https://onedrive.coca-cola.com/:f:/g/personal/yminagawa_coca-cola_com/ElC-IVU5k3lMkiOCgn3i0g4Bf4EHTWxssur6E5UIkgoC0g?e=xrMlXX
>
> Password: kafka
>
>
>
> All the best,
>
>
>
> [image: lingarogroup.com]
>
> *Roel Angeles*
>
> Consultant
>
>
>
> +63 9157163557
>
> roel.ange...@lingarogroup.com
>
> www.lingarogroup.com
>
>
>
>
>
>
>
> Lingaro Philippines Inc.
>
> 41F Philamlife Tower
>
> 8767 Paseo de Roxas
>
> Makati City, 1226
>
> +63 2753 8865
>
>
>
>
>
> *Next OOO planned:   *
>
>
>
>
>
> *From:* Adam Bellemare 
> *Sent:* Thursday, 19 August 2021 9:39 pm
> *To:* dev@kafka.apache.org
> *Cc:* Luke Chen ; Errol Pontillas <
> errol.pontil...@lingarogroup.com>; Jhunabelle Santos <
> jhunabelle.san...@lingarogroup.com>; Andrew Astrologo <
> andrew.astrol...@lingarogroup.com>; yminag...@coca-cola.com; coke.support
> 
> *Subject:* Re: Kafka Issue
>
>
>
> CAUTION: This email originated from outside of the Lingaro organization.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
> Hi folks
>
>
>
> You’ll need to host the images externally and send a link. The email
> server doesn’t send images due to the sheer number of recipients.
>
> Try using something like imgur
>
>
>
> Adam
>
>
>
> On Aug 19, 2021, at 2:18 AM, Roel Angeles 
> wrote:
>
> 
>
> Hi Luke,
>
>
>
> Resending the screenshot.
>
>
>
>
>
> Changes from
>
>
>
> to
>
>
>
>
>
>
>
> All the best,
>
>
>
> [image: lingarogroup.com]
>
> *Roel Angeles*
>
> Consultant
>
>
>
> +63 9157163557
>
> roel.ange...@lingarogroup.com
>
> www.lingarogroup.com
>
>
>
>
>
>
>
> Lingaro Philippines Inc.
>
> 41F Philamlife Tower
>
> 8767 Paseo de Roxas
>
> Makati City, 1226
>
> +63 2753 8865
>
>
>
>
>
> *Next OOO planned:   *
>
>
>
>
>
> *From:* Luke Chen 
> *Sent:* Thursday, 19 August 2021 2:44 pm
> *To:* dev 
> *Cc:* Roel Angeles ; Errol Pontillas <
> errol.pontil...@lingarogroup.com>; Jhunabelle Santos <
> jhunabelle.san...@lingarogroup.com>; Andrew Astrologo <
> andrew.astrol...@lingarogroup.com>; yminag...@coca-cola.com; coke.support
> 
> *Subject:* Re: Kafka Issue
>
>
>
> CAUTION: This email originated from outside of the Lingaro organization.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
> Hi Roel,
>
> I'm not sure if it's just me or not, but I can't see your screenshots, so
> I don't know what happened.
>
>
>
> Thanks.
>
> Luke
>
>
>
> On Wed, Aug 18, 2021 at 10:12 PM Jason Kamdon <
> jason.kam...@lingarogroup.com> wrote:
>
> Hi Team,
>
>
>
> Can we have an update regarding below concern.
>
>
>
>
>
> Thanks!
>
>
>
> Regards,
>
> *Jason Kamdon*
>
> Service Level Manager
>
>  +639178544422
>
> jason.kam...@lingarogroup.com
>
> www.lingarogroup.com
>
>
>
>
>
>
>
> Lingaro Philippines Inc.
>
> 41F Philamlife Tower
>
> 8767 Paseo de Roxas
>
> Makati City, 1226
>
> +63 2753 8865
>
>
>
>
>
> *Next OOO planned:*
>
> *Privacy notice:* Your personal data is being administered by Lingaro sp.
> z o.o., with its registered office in Warsaw under Polish National Court
> Register (KRS) number: 241638 ("Lingaro"). Lingaro processes your
> personal data for the purpose of contacting you and to perform its services
> on the basis of a contract (as per Article 6 sec. 1(b) GDPR: in connection
> with performing a contract or to undertake the activities upon your request
> prior to the conclusion of a contract), on the basis of legitimate interest
> of Lingaro (Article 6 sec. 1(f) GDPR) and/or on the basis of established
> provisions of law (

[VOTE] KIP-766: fetch/findSessions queries with open endpoints for SessionStore/WindowStore

[Luke Chen]

Hi all,
I'd like to start to vote for
*KIP-766: fetch/findSessions queries with open endpoints for
WindowStore/SessionStore*.

This is a follow-up KIP for KIP-763: Range queries with open endpoints
.
In KIP-763, we focused on *ReadOnlyKeyValueStore*, in this KIP, we'll focus
on *ReadOnlySessionStore* and *ReadOnlyWindowStore, *to have open endpoints
queries for SessionStore/WindowStore.

The KIP can be found here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186876596

Thank you.
Luke

Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 docker image

[Luke Chen]

Hi Ashish,
I suggested that you upgrade to V2.8.
I checked 2 of the CVEs, and are fixed (or not used, like libfetch) in
V2.8.
If you still found the CVEs existed in V2.8, please raise it.

Thank you.
Luke




On Wed, Sep 1, 2021 at 4:07 AM Ashish Patil  wrote:

> Hi Team
>
> I wanted to use the 2.6.0 docker image for Kafka but It has lots of
> security vulnerabilities.
> Please find the below list of security vulnerabilities
> **
> CVE-2021-36159
> CVE-2020-25649 
> CVE-2021-22926
> CVE-2021-22922
> CVE-2021-22924
> CVE-2021-22922
> CVE-2021-22924
> CVE-2021-31535
> CVE-2019-17571 
> **
>
> I did raise this issue here
> https://github.com/wurstmeister/kafka-docker/issues/681 but it looks like
> the issue is within the Kafka binary.
>
> Do we have any plan to fix this in the coming version or any suggestions
> around this?
>
> Thanks
>
> Ashish
>

Re: [VOTE] KIP-748: Add Broker Count Metrics (restarted)

[Luke Chen]

+1 (non-binding)
Thanks for the KIP!

Thank you.
Luke

On Wed, Aug 25, 2021 at 4:55 AM Jason Gustafson 
wrote:

> Thanks Colin! +1
>
> On Mon, Aug 23, 2021 at 1:37 PM David Arthur  wrote:
>
> > Thanks, Colin. Looks good to me!
> >
> > +1 binding
> >
> > -David
> >
> > On Mon, Aug 16, 2021 at 3:37 PM David Jacot 
> wrote:
> >
> > > Hi Colin,
> > >
> > > Thanks for restarting this KIP.
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > David
> > >
> > > Le lun. 16 août 2021 à 21:32, Colin McCabe  a
> écrit
> > :
> > >
> > > > Hi all,
> > > >
> > > > Two months ago we tried to vote on this minor KIP adding two new
> broker
> > > > metrics. However, things got busy and we weren't able to complete the
> > > vote.
> > > > Now that things have quieted down a bit, let's restart the vote!
> > > >
> > > > Note that I made some minor changes to the KIP after some offline
> > > > discussion with Ryan Dielhenn and David Jacot. Specifically, I added
> a
> > > > metric for the number of fenced brokers.
> > > >
> > > > Here's the KIP:
> > > > https://cwiki.apache.org/confluence/x/N4rOCg
> > > >
> > > > Here's the discussion thread:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r308364dfbb3020e6151cef47237c28a4a540e187b8af84ddafec83af%40%3Cdev.kafka.apache.org%3E
> > > >
> > > > Please take a look if you have a chance!
> > > >
> > > > best,
> > > > Colin
> > > >
> > >
> >
> >
> > --
> > David Arthur
> >
>

Re: Failed Unit testings in ReassignPartitionsIntegrationTest

[Luke Chen]

Hi Yanwen,

Welcome to Kafka!
Usually, we posted flaky tests on JIRA when we saw failed tests on Jenkins
build, either on trunk build, or PR build.
What you mentioned 2 (flaky) failed tests have not failed in Jenkins for a
long time (since build # 364, check below link). Besides, it also passed in
my local environment. So, it should be the issue in your environment.

history for ReassignPartitionsIntegrationTest on jenkins:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/413/testReport/kafka.admin/ReassignPartitionsIntegrationTest/history/
[image: 圖片.png]

I don't know what suggestion I can provide, since it looks like you can
compile it successfully, and run tests successfully (mostly).
Do you have another environment to test? VM? Docker container?

I can have a quick con-call with you to see what might be wrong in your env
if you want. Just let me know.

Thank you.
Luke

On Thu, Aug 19, 2021 at 1:30 PM Yanwen Lin  wrote:

> Hi folks,
>
> I just joined Kafka community. I would like to work on a beginner issue
> but get blocked on running unit testings:
> I found that I have issue with running the following two unit testings
> even w/ a fresh git clone of the Kafka repo on trunk branch:
> ReassignPartitionsIntegrationTest. testReassignment() => I never passed
> this test
> ReassignPartitionsIntegrationTest. testReassignmentWithAlterIsrDisabled()
> ==> This is a flaky test, sometimes I can pass but sometimes not.
> I ran the following Gradle command:
>
> $./gradlew core:test --tests ReassignPartitionsIntegrationTest
>
> My Java version is:
> $ java -version
> java version "1.8.0_172-ea"
> Java(TM) SE Runtime Environment (build 1.8.0_172-ea-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.172-b03, mixed mode)
>
> My Scala version is:
> $ scala -version
> Scala code runner version 2.13.1 -- Copyright 2002-2019, LAMP/EPFL and
> Lightbend, Inc.
>
> I did a search in the Jira tickets and there is only one I found related
> to this issue: https://issues.apache.org/jira/browse/KAFKA-12933 <
> https://issues.apache.org/jira/browse/KAFKA-12933>.
>
> Does anyone encounter this issue before? Thanks!
>
> Best,
> Yanwen

Re: Kafka Issue

[Luke Chen]

Hi Roel,
I'm not sure if it's just me or not, but I can't see your screenshots, so I
don't know what happened.

Thanks.
Luke

On Wed, Aug 18, 2021 at 10:12 PM Jason Kamdon 
wrote:

> Hi Team,
>
>
>
> Can we have an update regarding below concern.
>
>
>
>
>
> Thanks!
>
>
>
> Regards,
>
> [image: lingarogroup.com]
>
> *Jason Kamdon*
>
> Service Level Manager
>
>  +639178544422
>
> jason.kam...@lingarogroup.com
>
> www.lingarogroup.com
>
>
>
>
>
>
>
> Lingaro Philippines Inc.
>
> 41F Philamlife Tower
>
> 8767 Paseo de Roxas
>
> Makati City, 1226
>
> +63 2753 8865
>
>
>
>
>
> *Next OOO planned:*
>
> *Privacy notice:* Your personal data is being administered by Lingaro sp.
> z o.o., with its registered office in Warsaw under Polish National Court
> Register (KRS) number: 241638 ("Lingaro"). Lingaro processes your
> personal data for the purpose of contacting you and to perform its services
> on the basis of a contract (as per Article 6 sec. 1(b) GDPR: in connection
> with performing a contract or to undertake the activities upon your request
> prior to the conclusion of a contract), on the basis of legitimate interest
> of Lingaro (Article 6 sec. 1(f) GDPR) and/or on the basis of established
> provisions of law (Article 6 sec. 1(c) GDPR) - depending on the
> circumstances. See the details on Lingaro website
> .
>
> [image: lingarogroup.com]
>
>
>
>
>
>
>
>
>
> *From:* Roel Angeles 
> *Sent:* Monday, August 16, 2021 2:25 PM
> *To:* dev@kafka.apache.org
> *Cc:* Jason Kamdon ; Errol Pontillas <
> errol.pontil...@lingarogroup.com>; Jhunabelle Santos <
> jhunabelle.san...@lingarogroup.com>; Andrew Astrologo <
> andrew.astrol...@lingarogroup.com>
> *Subject:* Kafka Issue
>
>
>
> Hi Kafka Dev Team,
>
>
>
> We are requesting on your help about on our encountered issue about kafka.
>
> Please see below screenshot of error. It says partition does not exist but
> on our event hub it is active.
>
>
>
>
>
>
>
> Hoping for your kind response about this and big help for our team. Thanks.
>
>
>
> All the best,
>
>
>
> [image: lingarogroup.com]
>
> *Roel Angeles*
>
> Consultant
>
>
>
> +63 9157163557
>
> roel.ange...@lingarogroup.com
>
> www.lingarogroup.com
>
>
>
>
>
>
>
> Lingaro Philippines Inc.
>
> 41F Philamlife Tower
>
> 8767 Paseo de Roxas
>
> Makati City, 1226
>
> +63 2753 8865
>
>
>
>
>
> *Next OOO planned:   *
>
>
>
>
>

Re: Request contributor permission

[Luke Chen]

Hi Yanwen,
I think Matthias has already granted your permission to jira.
Please try to log in and see if you can assign a Kafka ticket to yourselves.

Thanks.
Luke

On Tue, Aug 17, 2021 at 11:15 AM Yanwen Lin  wrote:

> Hi Kafka team,
>
> Please help take a look. I’d like to contribute to Apache Kafka and my
> Jira ID is: ll1124278064
>
> Thanks!
>
> Best,
>
>
>
> > On Aug 14, 2021, at 12:05 AM, Yanwen Lin 
> wrote:
> >
> > Hi Kafka team,
> >
> > Can I ask for a contributor permission so that I can assign myself to
> the Kafka Jira ticket. My Jira id is: ll1124278064
> >
> > Thanks!
> >
> > Best,
> > Yanwen
>
>

Re: [EXTERNAL] Re: Security vulnerabilities in kafka:2.13-2.6.0/2.7.0 docker image

[Luke Chen]

Hi Ashish,

CVE-2021-36159: It's a libfetch lib vulnerability. It's not Kafka's
dependency lib. I guess it's the docker's base OS image.
CVE-2019-17571: a log4j vulnerability. KAFKA-9366
<https://issues.apache.org/jira/browse/KAFKA-9366> is working on it.

Thank you.
Luke

On Wed, Sep 1, 2021 at 9:26 PM Ashish Patil  wrote:

> Hi Team
>
>
>
> I tried upgrading it to 2.13_2.8.0 but still have these vulnerabilities.
>
>
>
>
>
> What is your suggestion on this?
>
>
>
> Thanks
>
> Ashish
>
>
>
> *From:* Jake Murphy Smith 
> *Sent:* 01 September 2021 09:31
> *To:* Ashish Patil 
> *Subject:* RE: [EXTERNAL] Re: Security vulnerabilities in
> kafka:2.13-2.6.0/2.7.0 docker image
>
>
>
>
>
>
>
> *From:* Luke Chen 
> *Sent:* 01 September 2021 04:11
> *To:* Kafka Users 
> *Cc:* dev@kafka.apache.org; Jake Murphy Smith 
> *Subject:* [EXTERNAL] Re: Security vulnerabilities in
> kafka:2.13-2.6.0/2.7.0 docker image
>
>
>
> *ATTENTION:* This email originated from outside of GM.
>
>
>
>
> Hi Ashish,
>
> I suggested that you upgrade to V2.8.
>
> I checked 2 of the CVEs, and are fixed (or not used, like libfetch) in
> V2.8.
>
> If you still found the CVEs existed in V2.8, please raise it.
>
>
>
> Thank you.
>
> Luke
>
>
>
>
>
>
>
>
>
> On Wed, Sep 1, 2021 at 4:07 AM Ashish Patil  wrote:
>
> Hi Team
>
> I wanted to use the 2.6.0 docker image for Kafka but It has lots of
> security vulnerabilities.
> Please find the below list of security vulnerabilities
> **
> CVE-2021-36159
> CVE-2020-25649 <https://github.com/advisories/GHSA-288c-cq4h-88gq>
> CVE-2021-22926
> CVE-2021-22922
> CVE-2021-22924
> CVE-2021-22922
> CVE-2021-22924
> CVE-2021-31535
> CVE-2019-17571 <https://github.com/advisories/GHSA-2qrg-x229-3v8q>
> **
>
> I did raise this issue here
> https://github.com/wurstmeister/kafka-docker/issues/681 but it looks like
> the issue is within the Kafka binary.
>
>
>
> Do we have any plan to fix this in the coming version or any suggestions
> around this?
>
> Thanks
>
> Ashish
>
>

Re: [DISCUSS] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

[Luke Chen]

Thanks for the KIP. Overall LGTM.

Just one thought, if we "rename" the config directly as mentioned in the
KIP, would that break existing applications?
Should we deprecate the old one first, and make the old/new names co-exist
for some period of time?

Public Interfaces

   - Adding a new config *input.buffer.max.bytes *applicable at a topology
   level. The importance of this config would be *Medium*.
   - Renaming *cache.max.bytes.buffering* to *statestore.cache.max.bytes*.



Thank you.
Luke

On Thu, Sep 2, 2021 at 1:50 AM Guozhang Wang  wrote:

> Currently the state store cache size default value is 10MB today, which
> arguably is rather small. So I'm thinking maybe for this config default to
> 512MB.
>
> Other than that, LGTM.
>
> On Sat, Aug 28, 2021 at 11:34 AM Sagar  wrote:
>
> > Thanks Guozhang and Sophie.
> >
> > Yeah a small default value would lower the throughput. I didn't quite
> > realise it earlier. It's slightly hard to predict this value so I would
> > guess around 1/2 GB to 1 GB? WDYT?
> >
> > Regarding the renaming of the config and the new metric, sure would
> include
> > it in the KIP.
> >
> > Lastly, importance would also. be added. I guess Medium should be ok.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Sat, Aug 28, 2021 at 10:42 AM Sophie Blee-Goldman
> >  wrote:
> >
> > > 1) I agree that we should just distribute the bytes evenly, at least
> for
> > > now. It's simpler to understand and
> > > we can always change it later, plus it makes sense to keep this aligned
> > > with how the cache works today
> > >
> > > 2) +1 to being conservative in the generous sense, it's just not
> > something
> > > we can predict with any degree
> > > of accuracy and even if we could, the appropriate value is going to
> > differ
> > > wildly across applications and use
> > > cases. We might want to just pick some multiple of the default cache
> > size,
> > > and maybe do some research on
> > > other relevant defaults or sizes (default JVM heap, size of available
> > > memory in common hosts eg EC2
> > > instances, etc). We don't need to worry as much about erring on the
> side
> > of
> > > too big, since other configs like
> > > the max.poll.records will help somewhat to keep it from exploding.
> > >
> > > 4) 100%, I always found the *cache.max.bytes.buffering* config name to
> be
> > > incredibly confusing. Deprecating this in
> > > favor of "*statestore.cache.max.bytes*" and aligning it to the new
> input
> > > buffer config sounds good to me to include here.
> > >
> > > 5) The KIP should list all relevant public-facing changes, including
> > > metadata like the config's "Importance". Personally
> > > I would recommend Medium, or even High if we're really worried about
> the
> > > default being wrong for a lot of users
> > >
> > > Thanks for the KIP, besides those few things that Guozhang brought up
> and
> > > the config importance, everything SGTM
> > >
> > > -Sophie
> > >
> > > On Thu, Aug 26, 2021 at 2:41 PM Guozhang Wang 
> > wrote:
> > >
> > > > 1) I meant for your proposed solution. I.e. to distribute the
> > configured
> > > > bytes among threads evenly.
> > > >
> > > > 2) I was actually thinking about making the default a large enough
> > value
> > > so
> > > > that we would not introduce performance regression: thinking about a
> > use
> > > > case with many partitions and each record may be large, then
> > effectively
> > > we
> > > > would only start pausing when the total bytes buffered is pretty
> large.
> > > If
> > > > we set the default value to small, we would be "more aggressive" on
> > > pausing
> > > > which may impact throughput.
> > > >
> > > > 3) Yes exactly, this would naturally be at the "partition-group"
> class
> > > > since that represents the task's all input partitions.
> > > >
> > > > 4) This is just a bold thought, I'm interested to see other's
> thoughts.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Aug 23, 2021 at 4:10 AM Sagar 
> > wrote:
> > > >
> > > > > Thanks Guozhang.
> > > > >
> > > > > 1) Just for my confirmation, when you say we should proceed with
> the
> > > even
> > > > > distribution of bytes, are you referring to the Proposed Solution
> in
> > > the
> > > > > KIP or the option you had considered in the JIRA?
> > > > > 2) Default value for the config is something that I missed. I agree
> > we
> > > > > can't have really large values as it might be detrimental to the
> > > > > performance. Maybe, as a starting point, we assume that only 1
> Stream
> > > > Task
> > > > > is running so what could be the ideal value in such a scenario?
> > > Somewhere
> > > > > around 10MB similar to the caching config?
> > > > > 3) When you say,  *a task level metric indicating the current
> totally
> > > > > aggregated metrics, * you mean the bytes aggregated at a task
> level?
> > > > > 4) I am ok with the name change, but would like to know others'
> > > thoughts.
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > > On Sun, Aug 22, 2021 at 11:54 

Re: [VOTE] KIP-761: Add Total Blocked Time Metric to Streams

[Luke Chen]

Thanks for the KIP.

+1 (non-binding)

Thank you.
Luke

On Wed, Sep 1, 2021 at 2:36 AM Guozhang Wang  wrote:

> Thanks for letting us know, Rohan.
>
> On Tue, Aug 31, 2021 at 1:08 AM Rohan Desai 
> wrote:
>
> > FYI I've updated the metric names in the KIP to the form
> ".*-time-ns-total"
> > and clarified that the times being measured are in nanoseconds.
> >
> > On Wed, Jul 21, 2021 at 5:09 PM Rohan Desai 
> > wrote:
> >
> > > Now that the discussion thread's been open for a few days, I'm calling
> > for
> > > a vote on
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-761%3A+Add+Total+Blocked+Time+Metric+to+Streams
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-773 Differentiate consistently metric latency measured in millis and nanos

[Luke Chen]

Thanks for the KIP.

+1 (non-binding)

Thanks.
Luke

On Sat, Sep 4, 2021 at 10:32 AM Guozhang Wang  wrote:

> Thanks Josep,
>
> Took a look at the KIP, LGTM.
>
> On Fri, Sep 3, 2021 at 11:25 AM Josep Prat 
> wrote:
>
> > Hi there,
> >
> > Since it's a rather small KIP, I'd like to start a vote for the KIP-773
> > Differentiate consistently metric latency measured  in millis and nanos.
> > The KIP page The KIP can be found at:
> > https://cwiki.apache.org/confluence/x/ZwNACw
> >
> > Thanks in advance,
> >
> > --
> >
> > Josep Prat
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491715557497
> >
> > *w:* aiven.io
> >
> > *e:* josep.p...@aiven.io
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

[Luke Chen]

Thanks for the KIP.

+ 1 (non-binding)

Thanks.
Luke

On Wed, Sep 8, 2021 at 2:48 PM Josep Prat 
wrote:

> +1 (non binding).
>
> Thanks for the KIP Sagar!
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Wed, Sep 8, 2021, 03:29 Sophie Blee-Goldman  >
> wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP!
> >
> > -Sophie
> >
> > On Tue, Sep 7, 2021 at 1:59 PM Guozhang Wang  wrote:
> >
> > > Thanks Sagar, +1 from me.
> > >
> > >
> > > Guozhang
> > >
> > > On Sat, Sep 4, 2021 at 10:29 AM Sagar 
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > I would like to start a vote on the following KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186878390
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.1.0 release

[Luke Chen]

Thanks for volunteering!
+1 (non-binding)

Look forward to V3.1.0!

Thank you.
Luke

On Sat, Sep 18, 2021 at 8:55 PM Israel Ekpo  wrote:

> Thanks for volunteering David. It’s great that we are already planning the
> 3.1.0 release.
>
>
> On Thu, Sep 16, 2021 at 3:38 PM Bill Bejeck  wrote:
>
> > Thanks for volunteering for the 3.1.0 release David!
> >
> > It's a +1 from me.
> >
> > -Bill
> >
> > On Thu, Sep 16, 2021 at 3:08 PM Konstantine Karantasis <
> > kkaranta...@apache.org> wrote:
> >
> > > Thanks for volunteering to run 3.1.0 David!
> > >
> > > +1
> > >
> > > Konstantine
> > >
> > >
> > > On Thu, Sep 16, 2021 at 6:42 PM Ismael Juma  wrote:
> > >
> > > > +1, thanks for volunteering David!
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Sep 16, 2021, 6:47 AM David Jacot
>  > >
> > > > wrote:
> > > >
> > > > > Hello All,
> > > > >
> > > > > I'd like to volunteer to be the release manager for our next
> > > > > feature release, 3.1.0. If there are no objections, I'll send
> > > > > out the release plan soon.
> > > > >
> > > > > Regards,
> > > > > David
> > > > >
> > > >
> > >
> >
>

[DISCUSS] KIP-776: Add Consumer#peek for debugging/tuning

[Luke Chen]

Hi everyone,

I'd like to discuss the following proposal to add Consumer#peek for
debugging/tuning.

The main purpose for Consumer#peek is to allow users:

   1. peek what records existed at broker side and not increasing the
   position offsets.
   2. throw exceptions when there is connection error existed between
   consumer and broker (or other exceptions will be thrown by "poll")


detailed description can be found her:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188746244


Any comments and feedback are welcomed.

Thank you.
Luke

Re: Contribution help please

[Luke Chen]

Hi Jon,
I checked your PR. I think you didn't set the "indent" correctly.
It looks like your indent setting is to 2, but we use 4.
Please update the setting in IntelliJ.
REF: https://www.jetbrains.com/help/idea/reformat-and-rearrange-code.html

Thank you.
Luke

On Mon, Sep 20, 2021 at 2:47 PM Jon McEwen  wrote:

> Hello Kafka folks,
>
> Could somone please tell me how to correctly format kafka Java code?
> Either on command line or in IntelliJ.
>
> I'm working on KAFKA-13303.
>
> Many thanks
>
> Jon McEwen
>
> https://github.com/jonmcewen
>

Re: [DISCUSS] KIP-766: fetch/findSessions queries with open endpoints for SessionStore/WindowStore

[Luke Chen]

Hi all,
Sorry that I found the KIP link I provided in the previous email is wrong.
Updated link is as below.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186876596

Thank you.
Luke


On Thu, Aug 5, 2021 at 2:31 PM Luke Chen  wrote:

> Hi everyone,
>
> I'd like to start the discussion for *KIP-766: fetch/findSessions queries
> with open endpoints for WindowStore/SessionStore*.
>
> This is a follow-up KIP for KIP-763: Range queries with open endpoints
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints>.
> In KIP-763, we focused on *ReadOnlyKeyValueStore*, in this KIP, we'll
> focus on *ReadOnlySessionStore* and *ReadOnlyWindowStore, *to have open
> endpoints queries for SessionStore/WindowStore.
>
> The KIP can be found here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186876596
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-763%3A+Range+queries+with+open+endpoints>
>
> Thank you.
> Luke
>
>

[DISCUSS] KIP-766: fetch/findSessions queries with open endpoints for SessionStore/WindowStore

[Luke Chen]

Hi everyone,

I'd like to start the discussion for *KIP-766: fetch/findSessions queries
with open endpoints for WindowStore/SessionStore*.

This is a follow-up KIP for KIP-763: Range queries with open endpoints
.
In KIP-763, we focused on *ReadOnlyKeyValueStore*, in this KIP, we'll focus
on *ReadOnlySessionStore* and *ReadOnlyWindowStore, *to have open endpoints
queries for SessionStore/WindowStore.

The KIP can be found here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186876596


Thank you.
Luke

Re: [DISCUSS] KIP-780: Support fine-grained compression options

[Luke Chen]

Hi Dongjin,
Thanks for the KIP, and the benchmark results. It makes sense to me.

Just one question:
> compression.zstd.window: enables long mode; the log of the window size
that zstd uses to memorize the compressing data. (available: [10, 22],
default: 0 (disables long mode.))

It said, available value is [10, 22], but default is a value out of that
range, which should be wrong.

Thank you.
Luke

On Sun, Oct 10, 2021 at 9:50 PM Dongjin Lee  wrote:

> Hi Kafka dev,
>
> I would like to start the discussion of KIP-780: Support fine-grained
> compression options.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options
>
> Here is some context or history on this feature; initially, this feature
> was intended to be a part of KIP-390: Support Compression Level, but when I
> was working on it, I could not find the evidence that these options can
> improve the performance, so it was excluded from the final proposal. Since
> this (tentative) conclusion was somewhat strange, KIP-390 was passed under
> the condition that a following work should be done for the
> buffer/block/window-related configuration options.
>
> And after some repetitive prototypes and benchmarks, it seems like I
> finally found the evidence. It is why I am submitting it as a separate
> proposal now. The document also includes what I found during the tests in
> the Benchmark section.
>
> All kinds of feedbacks are greatly appreciated!
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: Why need handle delete topic in topic change event

[Luke Chen]

Hi 晓兵,
I know what you mean now.
Yes, that makes sense to me, as long as you confirmed that each topic
deletion, we'll put a znode under "delete_topics".
Please open a jira ticket and welcome to submit PR.

Thank you.
Luke

On Tue, Oct 12, 2021 at 10:32 AM 方晓兵 <94fxiaob...@gmail.com> wrote:

> Hi Luke,
> What I mean is whether there is no need to listen to the topic znode
> deletion event, because `controllerContext#removeTopic` has been actively
> called after deleting znode in
> `TopicDeletionManager.completeDeleteTopic()`. Is it better for
> TopicChangeHandler to only handle child add events?
>
> > 2021年10月12日 上午10:23，Luke Chen  写道：
> >
> > Hi 晓兵,
> > Are you saying we should not call `removeTopic` because the topic znode
> is
> > deleted?
> > Have a quick look at the `controllerContext#removeTopic` implementation,
> it
> > looks like we only clean up the metrics, and local maps. Why is it a
> > problem we called it here? Does it caused any error?
> > Could you elaborate it more?
> >
> > Thank you.
> > Luke
> >
> > On Mon, Oct 11, 2021 at 6:10 PM 方晓兵 <94fxiaob...@gmail.com> wrote:
> >
> >> Hi Team,
> >>
> >> I have a problem when I study kafka code in version 2.8.0.
> >>
> >> I see `controllerContext.removeTopic(topic)` have been called in
> >> `TopicDeletionManager.completeDeleteTopic()`. TopicChangeHandler not
> only
> >> listen to child delete but also listen to child add. So every time a
> topic
> >> is deleted, an invalid event will be triggered because
> >> `controllerContext.removeTopic(topic)` have been called in
> >> `TopicDeletionManager.completeDeleteTopic()` after delete this topic
> >> zookeeper path. Is it code that needs to be optimized?
> >>
> >> Grateful and look forward to answers
>
>

Re: Why need handle delete topic in topic change event

[Luke Chen]

Hi 晓兵,
Are you saying we should not call `removeTopic` because the topic znode is
deleted?
Have a quick look at the `controllerContext#removeTopic` implementation, it
looks like we only clean up the metrics, and local maps. Why is it a
problem we called it here? Does it caused any error?
Could you elaborate it more?

Thank you.
Luke

On Mon, Oct 11, 2021 at 6:10 PM 方晓兵 <94fxiaob...@gmail.com> wrote:

> Hi Team,
>
> I have a problem when I study kafka code in version 2.8.0.
>
> I see `controllerContext.removeTopic(topic)` have been called in
> `TopicDeletionManager.completeDeleteTopic()`. TopicChangeHandler not only
> listen to child delete but also listen to child add. So every time a topic
> is deleted, an invalid event will be triggered because
> `controllerContext.removeTopic(topic)` have been called in
> `TopicDeletionManager.completeDeleteTopic()` after delete this topic
> zookeeper path. Is it code that needs to be optimized?
>
> Grateful and look forward to answers

Re: [DISCUSS] Apache Kafka 3.1.0 release

[Luke Chen]

Hi David,
KIP-766 is merged into trunk. Please help add it into the release plan.

Thank you.
Luke

On Mon, Oct 11, 2021 at 10:50 PM David Jacot 
wrote:

> Hi Michael,
>
> Sure. I have updated the release plan to include it. Thanks for the
> heads up.
>
> Best,
> David
>
> On Mon, Oct 11, 2021 at 4:39 PM Mickael Maison 
> wrote:
>
> > Hi David,
> >
> > You can add KIP-690 to the release plan. The vote passed months ago
> > and I merged the PR today.
> >
> > Thanks
> >
> > On Fri, Oct 8, 2021 at 8:32 AM David Jacot 
> > wrote:
> > >
> > > Hi folks,
> > >
> > > Just a quick reminder that KIP Freeze is next Friday, October 15th.
> > >
> > > Cheers,
> > > David
> > >
> > > On Wed, Sep 29, 2021 at 3:52 PM Chris Egerton
> > 
> > > wrote:
> > >
> > > > Thanks David!
> > > >
> > > > On Wed, Sep 29, 2021 at 2:56 AM David Jacot
> > 
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > Sure thing. I have added KIP-618 to the release plan. Thanks for
> the
> > > > heads
> > > > > up.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Wed, Sep 29, 2021 at 8:53 AM David Jacot 
> > wrote:
> > > > >
> > > > > > Hi Kirk,
> > > > > >
> > > > > > Yes, it is definitely possible if you can get the KIP voted
> before
> > the
> > > > > KIP
> > > > > > freeze
> > > > > > and the code committed before the feature freeze. Please, let me
> > know
> > > > > when
> > > > > > the
> > > > > > KIP is voted and I will add it to the release plan.
> > > > > >
> > > > > > Thanks,
> > > > > > David
> > > > > >
> > > > > > On Tue, Sep 28, 2021 at 7:05 PM Chris Egerton
> > > > > 
> > > > > > wrote:
> > > > > >
> > > > > >> Hi David,
> > > > > >>
> > > > > >> Wondering if we can get KIP-618 included? The vote passed months
> > ago
> > > > > and a
> > > > > >> PR has been available since mid-June.
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Chris
> > > > > >>
> > > > > >> On Tue, Sep 28, 2021 at 12:53 PM Kirk True <
> k...@mustardgrain.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > Hi David,
> > > > > >> >
> > > > > >> > Is it possible to try to get KIP-768 in 3.1? I have put it up
> > for a
> > > > > vote
> > > > > >> > and have much of it implemented already.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Kirk
> > > > > >> >
> > > > > >> > On Tue, Sep 28, 2021, at 3:11 AM, Israel Ekpo wrote:
> > > > > >> > > Ok. Sounds good, David.
> > > > > >> > >
> > > > > >> > > Let’s forge ahead. The plan looks good.
> > > > > >> > >
> > > > > >> > > On Tue, Sep 28, 2021 at 4:02 AM David Jacot
> > > > > >>  > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Israel,
> > > > > >> > > >
> > > > > >> > > > Yeah, 3.0 took quite a long time to be released. However,
> I
> > > > think
> > > > > >> > > > that we should stick to our time based release.
> > > > > >> > > >
> > > > > >> > > > Best,
> > > > > >> > > > David
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Tue, Sep 28, 2021 at 9:59 AM David Jacot <
> > > > dja...@confluent.io>
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Bruno,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for the heads up. I have removed it from the
> plan.
> > > > > >> > > > >
> > > > > >> > > > > Best,
> > > > > >> > > > > David
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Sep 27, 2021 at 11:04 AM Bruno Cadonna <
> > > > > >> cado...@apache.org>
> > > > > >> > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > >> Hi David,
> > > > > >> > > > >>
> > > > > >> > > > >> Thank you for the plan!
> > > > > >> > > > >>
> > > > > >> > > > >> KIP-698 will not make it for 3.1.0. Could you please
> > remove
> > > > it
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > >> plan?
> > > > > >> > > > >>
> > > > > >> > > > >> Best,
> > > > > >> > > > >> Bruno
> > > > > >> > > > >>
> > > > > >> > > > >> On 24.09.21 16:22, David Jacot wrote:
> > > > > >> > > > >> > Hi all,
> > > > > >> > > > >> >
> > > > > >> > > > >> > I just published a release plan here:
> > > > > >> > > > >> >
> > > > > >> >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.1.0
> > > > > >> > > > >> >
> > > > > >> > > > >> > The plan suggests the following dates:
> > > > > >> > > > >> >
> > > > > >> > > > >> > KIP Freeze: 15 October 2021
> > > > > >> > > > >> > Feature Freeze: 29 October 2021
> > > > > >> > > > >> > Code Freeze: 12 November 2021
> > > > > >> > > > >> >
> > > > > >> > > > >> > At least two weeks of stabilization will follow Code
> > > > Freeze.
> > > > > >> > > > >> >
> > > > > >> > > > >> > I have included all the currently approved KIPs
> > targeting
> > > > > >> 3.1.0.
> > > > > >> > > > Please
> > > > > >> > > > >> > let me know if I should add/remove any to/from the
> > plan.
> > > > > >> > > > >> >
> > > > > >> > > > >> > Please let me know if you have any objections.
> > > > > >> > > > >> >
> > > > > >> > > > >> > Regards,
> > > > > >> > > > >> > David
> > > > > >> > > > >> >
> > > > > >> > > > >> > On Mon, Sep 20, 

[DISCUSS] KIP-782: Expandable batch size in producer

[Luke Chen]

Hi Kafka dev,
I'd like to start the discussion for the proposal: KIP-782: Expandable
batch size in producer.

The main purpose for this KIP is to have better memory usage in producer,
and also save users from the dilemma while setting the batch size
configuration. After this KIP, users can set a higher batch.size without
worries, and of course, with an appropriate "batch.initial.size" and
"batch.reallocation.factor".

Derailed description can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer

Any comments and feedback are welcome.

Thank you.
Luke

Re: [DISCUSS] KIP-782: Expandable batch size in producer

[Luke Chen]

Hi Ismael and all devs,
Is there any comments/suggestions to this KIP?
If no, I'm going to update the KIP based on my previous mail, and start a
vote tomorrow or next week.

Thank you.
Luke

On Mon, Oct 18, 2021 at 2:40 PM Luke Chen  wrote:

> Hi Ismael,
> Thanks for your comments.
>
> 1. Why do we have to reallocate the buffer? We can keep a list of buffers
> instead and avoid reallocation.
> -> Do you mean we allocate multiple buffers with "buffer.initial.size",
> and link them together (with linked list)?
> ex:
> a. We allocate 4KB initial buffer
> | 4KB |
>
> b. when new records reached and the remaining buffer is not enough for the
> records, we create another batch with "batch.initial.size" buffer
> ex: we already have 3KB of data in the 1st buffer, and here comes the 2KB
> record
>
> | 4KB (1KB remaining) |
> now, record: 2KB coming
> We fill the 1st 1KB into 1st buffer, and create new buffer, and linked
> together, and fill the rest of data into it
> | 4KB (full) | ---> | 4KB (3KB remaining) |
>
> Is that what you mean?
> If so, I think I like this idea!
> If not, please explain more detail about it.
> Thank you.
>
> 2. I think we should also consider tweaking the semantics of batch.size so
> that the sent batches can be larger if the batch is not ready to be sent
> (while still respecting max.request.size and perhaps a new max.batch.size).
>
> --> In the KIP, I was trying to make the "batch.size" as the upper bound
> of the batch size, and introduce a "batch.initial.size" as initial batch
> size.
> So are you saying that we can let "batch.size" as initial batch size and
> introduce a "max.batch.size" as upper bound value?
> That's a good suggestion, but that would change the semantics of
> "batch.size", which might surprise some users. I think my original proposal
> ("batch.initial.size") is safer for users. What do you think?
>
> Thank you.
> Luke
>
>
> On Mon, Oct 18, 2021 at 3:12 AM Ismael Juma  wrote:
>
>> I think we should also consider tweaking the semantics of batch.size so
>> that the sent batches can be larger if the batch is not ready to be sent
>> (while still respecting max.request.size and perhaps a new
>> max.batch.size).
>>
>> Ismael
>>
>> On Sun, Oct 17, 2021, 12:08 PM Ismael Juma  wrote:
>>
>> > Hi Luke,
>> >
>> > Thanks for the KIP. Why do we have to reallocate the buffer? We can
>> keep a
>> > list of buffers instead and avoid reallocation.
>> >
>> > Ismael
>> >
>> > On Sun, Oct 17, 2021, 2:02 AM Luke Chen  wrote:
>> >
>> >> Hi Kafka dev,
>> >> I'd like to start the discussion for the proposal: KIP-782: Expandable
>> >> batch size in producer.
>> >>
>> >> The main purpose for this KIP is to have better memory usage in
>> producer,
>> >> and also save users from the dilemma while setting the batch size
>> >> configuration. After this KIP, users can set a higher batch.size
>> without
>> >> worries, and of course, with an appropriate "batch.initial.size" and
>> >> "batch.reallocation.factor".
>> >>
>> >> Derailed description can be found here:
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer
>> >>
>> >> Any comments and feedback are welcome.
>> >>
>> >> Thank you.
>> >> Luke
>> >>
>> >
>>
>

Re: [SPAM] Re: Why does Kafka have a higher throughput than Redis?

[Luke Chen]

Hi Vitor,
I'm not the expert, either, but I think Andrew's answer is pretty much the
reasons why Kafka is doing good.
And I'm not too familiar with Redis, either. But I'd say, there are many
configurations in each product to increase the throughput, and the use
cases are different, the comparison might not be fair.

For your question:
3.  Didn't know about this zero-copy technique, I'll read more about that
but feels like the result would be a response similar to as if kafka had
the info stored in-memory (as redis do) but that would still make me
question how is that Kafka can handle a higher throughput if the "design"
is so similar.
--> Again, I'm not familiar with Redis, but even you store data in memory,
if there's no OS's help, you still need to copy data to kernel space to
send to the receiver, compared with the zero-copy technique, all data flow
are within kernel space.

But again, the use cases are different, the comparison might not be fair.
We can only analyze and learn why and how they have good throughput.
That's my two cents.

Thank you.
Luke

On Fri, Oct 15, 2021 at 3:47 AM Vitor Augusto de Medeiros <
v.medei...@aluno.ufabc.edu.br> wrote:

> Thanks for the response, Andrew, i appreciate the help!
>
> Just i few thoughts that came up while reading your points:
>
>
>   1.  In theory, Redis is also handling/storing data in memory which makes
> me wonder why is that Kafka does it better? Perhaps it has to do with the
> API contract, where, as you said, there's no complex transactional software
> that might hurt performance.
>   2.  Didn't know there was such a big difference from linear to random
> writes, pretty awesome! But I still don't understand how disk usage, even
> If doing linear writes, is still allowing a throughput rate of 2 to 3x the
> amount of Redis, which doesn't use disk write/read at all and keep messages
> stored in memory.
>   3.  Didn't know about this zero-copy technique, I'll read more about
> that but feels like the result would be a response similar to as if kafka
> had the info stored in-memory (as redis do) but that would still make me
> question how is that Kafka can handle a higher throughput if the "design"
> is so similar.
>
>
> 
> De: Andrew Grant 
> Enviado: quinta-feira, 14 de outubro de 2021 15:55
> Para: dev@kafka.apache.org 
> Assunto: [SPAM] Re: Why does Kafka have a higher throughput than Redis?
>
> Hi Vitor,
>
> I'm not an expert and probably some more knowledgeable folks can also chime
> in (and correct me) but a few things came to mind:
>
> 1) On the write side (i.e. when using the publisher), Kafka does not flush
> data to disk by default. It writes to the page cache so all writes are sort
> of in-memory in a way. They're staged in the page cache and the kernel
> flushes the data asynchronously. Also the API contract for Kafka is quite
> "simple" in that it mostly reads and writes arbitrary sequences of bytes -
> there isn't as much complex transactional software in front of the
> writing/reading that might hurt performance compared to some other data
> stores. Note, Kafka does provide things like idempotence and transactions
> so it's not like there is never any overhead to consider.
>
> 2) Kafka reads and writes are conducive to being linear which helps a lot
> with performance. Random writes are a lot slower than linear ones.
>
> 3) For reading (i.e. when using the consumer) data Kafka uses a zero-copy
> technique in which data is directly sent from the page cache to the network
> buffer without going through user space which helps a lot.
>
> 4) Kafka batches aggressively.
>
> Here are two resources which might provide more information
> https://docs.confluent.io/platform/current/kafka/design.html,
>
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> .
>
> Hope this helps a bit.
>
> Andrew
>
> On Thu, Oct 14, 2021 at 1:11 PM Vitor Augusto de Medeiros <
> v.medei...@aluno.ufabc.edu.br> wrote:
>
> > Hi everyone,
> >
> >  i'm doing a benchmark comparison between Kafka and Redis for my final
> > bachelor paper and would like to understand more about why Kafka have
> > higher throughput if compared to Redis.
> >
> >  I noticed Redis has lower overall latency (and makes sense since it's
> > stored in memory) but cant figure out the difference in throughput.
> >
> > I found a study (not sure if i can post links here but it's named A
> > COMPARISON OF DATA INGESTION PLATFORMS IN REAL-TIME STREAM PROCESSING
> > PIPELINES by Sebastian Tallberg)
> > showing Kafka's throughput hitting 3x the amount of msg/s if compared to
> > Redis for a 1kB payload. I would like to understand what is in Kafka's
> > architecture that allows it to be a lot faster than other message
> > brokers/Redis in particular
> >
> > Thanks!
> >
>
>
> --
> Andrew Grant
> 8054482621
>

Re: Kafka client 2.7.1 missing JaasUtils.isZkSecurityEnabled() method

[Luke Chen]

Hi Alexandre
Yes, you're right. We renamed the `isZkSecurityEnabled` method name into
`isZkSaslEnabled`, because it checked sasl config only. You can check here

.

If you want to check TLS configuration, you can check here

and here
,
basically it just check the tls is enabled and client socket/keystore is
set.

Thank you.
Luke


On Thu, Oct 14, 2021 at 11:04 PM Alexandre Vermeerbergen <
avermeerber...@gmail.com> wrote:

> Hello,
>
> When upgrading from Kafka client 2.4.0 our dependencies to Kafka
> client 2.7.1, we noticed that JaasUtils.isZkSecurityEnabled() method
> no longer exists.
>
> Is there an equivalent method in Kafka client 2.7.1 which could use
> instead ?
>
> Kind regards,
> Alexandre
>

Re: [VOTE] KIP-764 Configurable backlog size for creating Acceptor

[Luke Chen]

Hi Okada,
Thanks for the KIP.
+1 (non-binding)

One thing to add is that you should add ServerSocket#bind java doc link
into the KIP.
I don't think everyone is familiar with the definition of the method
parameters.

Thank you.
Luke

On Mon, Oct 18, 2021 at 3:43 PM Haruki Okada  wrote:

> Hi Kafka.
>
> Let me bump this VOTE thread for the KIP.
> We applied proposed changes in the KIP to our large Kafka cluster by
> building patched Kafka internally and confirmed it's working well.
>
> Please feel free to give your feedback if there's any points to be
> clarified in the KIP.
>
> Thanks,
>
> 2021年8月9日(月) 11:25 Haruki Okada :
>
> > Thanks for your comment LI-san.
> >
> > Could anyone else review and vote for the KIP?
> >
> > I think the situation described in the KIP's motivation can happen in any
> > large-scale Kafka deployment, so may be helpful for many users while the
> > proposed changes are small enough.
> >
> >
> > Thanks,
> >
> > 2021年8月3日(火) 15:49 Xiangyuan LI :
> >
> >> Hi Haruki Okada:
> >>   i read your comment, thx for your detail explain!
> >>   add backlog parameter is a useful suggestion, hope it could added to
> >> kafka.
> >>
> >> Haruki Okada  于2021年8月2日周一 上午7:43写道：
> >>
> >> > Hi, Kafka.
> >> >
> >> > I would like to start a vote on KIP that makes SocketServer acceptor's
> >> > backlog size configurable.
> >> >
> >> > KIP:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-764%3A+Configurable+backlog+size+for+creating+Acceptor
> >> >
> >> > Discussion thread:
> >> >
> >> >
> >>
> https://lists.apache.org/thread.html/rd77469b7de0190d601dd37bd6894e1352a674d08038bcfe7ff68a1e0%40%3Cdev.kafka.apache.org%3E
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > 
> >> > Okada Haruki
> >> > ocadar...@gmail.com
> >> > 
> >> >
> >>
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>

Re: [VOTE] Add TaskId field to StreamsException

[Luke Chen]

Hi Sophie,
Add taskId to make the exception much clear is a good improvement.
+ 1 (non-binding)

Thank you.
Luke

On Mon, Oct 18, 2021 at 12:10 PM Sophie Blee-Goldman
 wrote:

> Hey all,
>
> I'd like to kick off the vote on this small KIP which adds a TaskId field
> to the StreamsException class. Please take a look and cast your vote when
> you have a chance.
>
> Links:
>
>- KIP-783: Add TaskId field to StreamsException
>
>- PR #11405 
>
>
> Thanks!
> Sophie
>

Re: [DISCUSS] KIP-782: Expandable batch size in producer

[Luke Chen]

Hi Ismael,
Thanks for your comments.

1. Why do we have to reallocate the buffer? We can keep a list of buffers
instead and avoid reallocation.
-> Do you mean we allocate multiple buffers with "buffer.initial.size", and
link them together (with linked list)?
ex:
a. We allocate 4KB initial buffer
| 4KB |

b. when new records reached and the remaining buffer is not enough for the
records, we create another batch with "batch.initial.size" buffer
ex: we already have 3KB of data in the 1st buffer, and here comes the 2KB
record

| 4KB (1KB remaining) |
now, record: 2KB coming
We fill the 1st 1KB into 1st buffer, and create new buffer, and linked
together, and fill the rest of data into it
| 4KB (full) | ---> | 4KB (3KB remaining) |

Is that what you mean?
If so, I think I like this idea!
If not, please explain more detail about it.
Thank you.

2. I think we should also consider tweaking the semantics of batch.size so
that the sent batches can be larger if the batch is not ready to be sent
(while still respecting max.request.size and perhaps a new max.batch.size).

--> In the KIP, I was trying to make the "batch.size" as the upper bound of
the batch size, and introduce a "batch.initial.size" as initial batch size.
So are you saying that we can let "batch.size" as initial batch size and
introduce a "max.batch.size" as upper bound value?
That's a good suggestion, but that would change the semantics of
"batch.size", which might surprise some users. I think my original proposal
("batch.initial.size") is safer for users. What do you think?

Thank you.
Luke


On Mon, Oct 18, 2021 at 3:12 AM Ismael Juma  wrote:

> I think we should also consider tweaking the semantics of batch.size so
> that the sent batches can be larger if the batch is not ready to be sent
> (while still respecting max.request.size and perhaps a new max.batch.size).
>
> Ismael
>
> On Sun, Oct 17, 2021, 12:08 PM Ismael Juma  wrote:
>
> > Hi Luke,
> >
> > Thanks for the KIP. Why do we have to reallocate the buffer? We can keep
> a
> > list of buffers instead and avoid reallocation.
> >
> > Ismael
> >
> > On Sun, Oct 17, 2021, 2:02 AM Luke Chen  wrote:
> >
> >> Hi Kafka dev,
> >> I'd like to start the discussion for the proposal: KIP-782: Expandable
> >> batch size in producer.
> >>
> >> The main purpose for this KIP is to have better memory usage in
> producer,
> >> and also save users from the dilemma while setting the batch size
> >> configuration. After this KIP, users can set a higher batch.size without
> >> worries, and of course, with an appropriate "batch.initial.size" and
> >> "batch.reallocation.factor".
> >>
> >> Derailed description can be found here:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer
> >>
> >> Any comments and feedback are welcome.
> >>
> >> Thank you.
> >> Luke
> >>
> >
>

Re: [DISCUSS] KIP-776: Add Consumer#peek for debugging/tuning

[Luke Chen]

Thanks for your feedback, Sagar, Boyang.

I've added an additional API to take the Set as the
partitions to fetch from. Good suggestion!
I also updated the java doc in the KIP.

And for the question that the behavior can also be achieved by using manual
offset commit + offset position rewind. That's true.
But I have the same thoughts as Sagar, which is that, it's for advanced
users.
Another reason is for simplicity. If you've ever used the peek API from
java collection (ex: Queue#peek), you should know what I'm talking about.
When you have data in a queue, if you want to know what the first data is
in the queue, you'd use peek(). You can also achieve it by remove() the 1st
element from queue, and then added it back to the right position, but I
believe that's not what you'd do.

Thank you.
Luke


On Tue, Sep 21, 2021 at 1:02 AM Sagar  wrote:

> Thanks Luke for the KIP. I think it makes sense.
>
> @Boyang,
>
> While it is possible to get the functionality using manual offset commit +
> offset position rewind as you stated, IMHO it could still be a very handy
> addition to the APIs.  The way I see it, manual offset commit + offset
> position rewind is for slightly more advanced users and the addition of
> peek() API would make it trivial to get the mentioned functionality.
>
> I agree to the point of adding a mechanism to fetch a more fine grained set
> of records. Maybe, add another API which takes a Set? In
> this case, we would probably need to add a behaviour to throw some
> exception when a user tries to peek from a TopicPartition that he/she isn't
> subscribed to.
>
> nit: In the javadoc, this line =>
>
> This method returns immediately if there are records available or
> exception thrown.
>
>
> should probably be =>
>
>
> This method returns immediately if there are no records available or
> exception thrown.
>
>
> Thanks!
>
> Sagar.
>
>
>
>
> On Mon, Sep 20, 2021 at 4:22 AM Boyang Chen 
> wrote:
>
> > Thanks Luke for the KIP.
> >
> > I think I understand the motivation is to avoid affecting offset
> positions
> > of the records, but the feature could be easily realized on the user side
> > by using manual offset commit + offset position rewind. So the new peek()
> > function doesn't provide any new functionality IMHO, weakening the
> > motivation a bit.
> >
> > Additionally, for the peek() case, I believe that users may want to have
> > more fine-grained exposure of records, such as from specific partitions
> > instead of getting random records. It's probably useful to define an
> option
> > handle class in the parameters to help clarify what specific records to
> be
> > returned.
> >
> > Boyang
> >
> > On Sun, Sep 19, 2021 at 1:51 AM Luke Chen  wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to discuss the following proposal to add Consumer#peek for
> > > debugging/tuning.
> > >
> > > The main purpose for Consumer#peek is to allow users:
> > >
> > >1. peek what records existed at broker side and not increasing the
> > >position offsets.
> > >2. throw exceptions when there is connection error existed between
> > >consumer and broker (or other exceptions will be thrown by "poll")
> > >
> > >
> > > detailed description can be found her:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188746244
> > >
> > >
> > > Any comments and feedback are welcomed.
> > >
> > > Thank you.
> > > Luke
> > >
> >
>

Re: [VOTE] KIP-774: Deprecate public access to Admin client's *Result constructors

[Luke Chen]

Hi Tom,
Thanks for the KIP.
I agree with you that we should change it back to non-public for future
enhancement.

+1 (non-binding)

Thank you.
Luke

On Mon, Sep 20, 2021 at 9:05 PM Josep Prat 
wrote:

> Hi Tom,
>
> Thanks for the KIP. It's a +1 (non binding) from my side.
>
> Best,
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Sep 20, 2021, 14:45 Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-774 which proposes to deprecate public
> > access to the Admin client's *Result constructors:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-774%3A+Deprecate+public+access+to+Admin+client%27s+*Result+constructors
> >
> > Thanks for your time,
> >
> > Tom
> >
>
> On Mon, Sep 20, 2021, 14:45 Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-774 which proposes to deprecate public
> > access to the Admin client's *Result constructors:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-774%3A+Deprecate+public+access+to+Admin+client%27s+*Result+constructors
> >
> > Thanks for your time,
> >
> > Tom
> >
>
> On Mon, Sep 20, 2021, 14:45 Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-774 which proposes to deprecate public
> > access to the Admin client's *Result constructors:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-774%3A+Deprecate+public+access+to+Admin+client%27s+*Result+constructors
> >
> > Thanks for your time,
> >
> > Tom
> >
>
> On Mon, Sep 20, 2021, 14:45 Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to start a vote for KIP-774 which proposes to deprecate public
> > access to the Admin client's *Result constructors:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-774%3A+Deprecate+public+access+to+Admin+client%27s+*Result+constructors
> >
> > Thanks for your time,
> >
> > Tom
> >
>

[VOTE] KIP-776: Add Consumer#peek for debugging/tuning

[Luke Chen]

Hi everyone,

I'd like to start a vote for the following proposal to add Consumer#peek
for debugging/tuning.

The main purpose for Consumer#peek is to allow users:

   1. peek what records existed at broker side and not increasing the
   position offsets.
   2. throw exceptions when there is connection error existed between
   consumer and broker (or other exceptions will be thrown by "poll")


Detailed description can be found here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188746244

Thank you
Luke

Re: [DISCUSS] KIP-776: Add Consumer#peek for debugging/tuning

[Luke Chen]

Hi Mickael,
Thanks for your comments.
I've added a use case into the KIP, and added Boyang's comment into
rejected alternatives section.

Hope that makes sense and strengthens the motivation.
If you have other suggestions, please let me know.

Thank you.
Luke

On Mon, Oct 4, 2021 at 10:23 PM Mickael Maison 
wrote:

> Hi Luke,
>
> Thanks for the KIP.
>
> Can you clarify the use cases you have in mind exactly? This would
> strengthen the motivation which I find a bit weak at the moment.
>
> As mentioned by Boyang, it's possible to achieve something similar
> with the existing APIs, for example with poll/seek or with
> listOffsets. Can you list them in the rejected alternatives section
> and explain why they were rejected.
>
> Thanks
>
>
> On Tue, Sep 21, 2021 at 9:47 AM Luke Chen  wrote:
> >
> > Thanks for your feedback, Sagar, Boyang.
> >
> > I've added an additional API to take the Set as the
> > partitions to fetch from. Good suggestion!
> > I also updated the java doc in the KIP.
> >
> > And for the question that the behavior can also be achieved by using
> manual
> > offset commit + offset position rewind. That's true.
> > But I have the same thoughts as Sagar, which is that, it's for advanced
> > users.
> > Another reason is for simplicity. If you've ever used the peek API from
> > java collection (ex: Queue#peek), you should know what I'm talking about.
> > When you have data in a queue, if you want to know what the first data is
> > in the queue, you'd use peek(). You can also achieve it by remove() the
> 1st
> > element from queue, and then added it back to the right position, but I
> > believe that's not what you'd do.
> >
> > Thank you.
> > Luke
> >
> >
> > On Tue, Sep 21, 2021 at 1:02 AM Sagar  wrote:
> >
> > > Thanks Luke for the KIP. I think it makes sense.
> > >
> > > @Boyang,
> > >
> > > While it is possible to get the functionality using manual offset
> commit +
> > > offset position rewind as you stated, IMHO it could still be a very
> handy
> > > addition to the APIs.  The way I see it, manual offset commit + offset
> > > position rewind is for slightly more advanced users and the addition of
> > > peek() API would make it trivial to get the mentioned functionality.
> > >
> > > I agree to the point of adding a mechanism to fetch a more fine
> grained set
> > > of records. Maybe, add another API which takes a Set?
> In
> > > this case, we would probably need to add a behaviour to throw some
> > > exception when a user tries to peek from a TopicPartition that he/she
> isn't
> > > subscribed to.
> > >
> > > nit: In the javadoc, this line =>
> > >
> > > This method returns immediately if there are records available or
> > > exception thrown.
> > >
> > >
> > > should probably be =>
> > >
> > >
> > > This method returns immediately if there are no records available or
> > > exception thrown.
> > >
> > >
> > > Thanks!
> > >
> > > Sagar.
> > >
> > >
> > >
> > >
> > > On Mon, Sep 20, 2021 at 4:22 AM Boyang Chen <
> reluctanthero...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Luke for the KIP.
> > > >
> > > > I think I understand the motivation is to avoid affecting offset
> > > positions
> > > > of the records, but the feature could be easily realized on the user
> side
> > > > by using manual offset commit + offset position rewind. So the new
> peek()
> > > > function doesn't provide any new functionality IMHO, weakening the
> > > > motivation a bit.
> > > >
> > > > Additionally, for the peek() case, I believe that users may want to
> have
> > > > more fine-grained exposure of records, such as from specific
> partitions
> > > > instead of getting random records. It's probably useful to define an
> > > option
> > > > handle class in the parameters to help clarify what specific records
> to
> > > be
> > > > returned.
> > > >
> > > > Boyang
> > > >
> > > > On Sun, Sep 19, 2021 at 1:51 AM Luke Chen  wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to discuss the following proposal to add Consumer#peek for
> > > > > debugging/tuning.
> > > > >
> > > > > The main purpose for Consumer#peek is to allow users:
> > > > >
> > > > >1. peek what records existed at broker side and not increasing
> the
> > > > >position offsets.
> > > > >2. throw exceptions when there is connection error existed
> between
> > > > >consumer and broker (or other exceptions will be thrown by
> "poll")
> > > > >
> > > > >
> > > > > detailed description can be found her:
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=188746244
> > > > >
> > > > >
> > > > > Any comments and feedback are welcomed.
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > >
> > >
>

Re: [VOTE] KIP-782: Expandable batch size in producer

[Luke Chen]

Hi Artem,
Thanks for your good suggestion again.
I've combined your idea into this KIP, and updated it.
Note, in the end, I still keep the "batch.initial.size" config (default is
0, which means "batch.size" will be initial batch size) for better memory
conservation.

Detailed description can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer

Let me know if you have other suggestions.

Thank you.
Luke

On Sat, Oct 23, 2021 at 10:50 AM Luke Chen  wrote:

> Hi Artem,
> Thanks for the suggestion. Let me confirm my understanding is correct.
> So, what you suggest is that the "batch.size" is more like a "soft limit"
> batch size, and the "hard limit" is "batch.max.size". When reaching the
> batch.size of the buffer, it means the buffer is "ready" to be be sent. But
> before the linger.ms reached, if there are more data coming, we can still
> accumulate it into the same buffer, until it reached the "batch.max.size".
> After it reached the "batch.max.size", we'll create another batch for it.
>
> So after your suggestion, we won't need the "batch.initial.size", and we
> can use "batch.size" as the initial batch size. We list each "batch.size"
> together, until it reached "batch.max.size". Something like this:
>
> [image: image.png]
> Is my understanding correct?
> If so, that sounds good to me.
> If not, please kindly explain more to me.
>
> Thank you.
> Luke
>
>
>
>
> On Sat, Oct 23, 2021 at 2:13 AM Artem Livshits
>  wrote:
>
>> Hi Luke,
>>
>> Nice suggestion.  It should optimize how memory is used with different
>> production rates, but I wonder if we can take this idea further and
>> improve
>> batching in general.
>>
>> Currently batch.size is used in two conditions:
>>
>> 1. When we append records to a batch in the accumulator, we create a new
>> batch if the current batch would exceed the batch.size.
>> 2. When we drain the batch from the accumulator, a batch becomes 'ready'
>> when it reaches batch.size.
>>
>> The second condition is good with the current batch size, because if
>> linger.ms is greater than 0, the send can be triggered by accomplishing
>> the
>> batching goal.
>>
>> The first condition, though, leads to creating many batches if the network
>> latency or production rate (or both) is high, and with 5 in-flight and
>> 16KB
>> batches we can only have 80KB of data in-flight per partition.  Which
>> means
>> that with 50ms latency, we can only push ~1.6MB/sec per partition (this
>> goes down if we consider higher latencies, e.g. with 100ms we can only
>> push
>> ~0.8MB/sec).
>>
>> I think it would be great to separate the two sizes:
>>
>> 1. When appending records to a batch, create a new batch if the current
>> exceeds a larger size (we can call it batch.max.size), say 256KB by
>> default.
>> 2. When we drain, consider batch 'ready' if it exceeds batch.size, which
>> is
>> 16KB by default.
>>
>> For memory conservation we may introduce batch.initial.size if we want to
>> have a flexibility to make it even smaller than batch.size, or we can just
>> always use batch.size as the initial size (in which case we don't
>> need batch.initial.size config).
>>
>> -Artem
>>
>> On Fri, Oct 22, 2021 at 1:52 AM Luke Chen  wrote:
>>
>> > Hi Kafka dev,
>> > I'd like to start a vote for the proposal: KIP-782: Expandable batch
>> size
>> > in producer.
>> >
>> > The main purpose for this KIP is to have better memory usage in
>> producer,
>> > and also save users from the dilemma while setting the batch size
>> > configuration. After this KIP, users can set a higher batch.size without
>> > worries, and of course, with an appropriate "batch.initial.size".
>> >
>> > Derailed description can be found here:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer
>> >
>> > Any comments and feedback are welcome.
>> >
>> > Thank you.
>> > Luke
>> >
>>
>

[VOTE] KIP-782: Expandable batch size in producer

[Luke Chen]

Hi Kafka dev,
I'd like to start a vote for the proposal: KIP-782: Expandable batch size
in producer.

The main purpose for this KIP is to have better memory usage in producer,
and also save users from the dilemma while setting the batch size
configuration. After this KIP, users can set a higher batch.size without
worries, and of course, with an appropriate "batch.initial.size".

Derailed description can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expandable+batch+size+in+producer

Any comments and feedback are welcome.

Thank you.
Luke

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Luke Chen]

Hi Colin,
Thanks for your comment.

> How are we going to avoid the situation where the broker restarts, and
the same generation number is reused?

Actually, this KIP doesn't have anything to do with the brokers. The
"generation" field I added, is in the subscription metadata, which will not
be deserialized by brokers. The metadata is only deserialized by consumer
lead. And for the consumer lead, the only thing the lead cared about, is
the highest generation of the ownedPartitions among all the consumers. With
the highest generation of the ownedPartitions, the consumer lead can
distribute the partitions as sticky as possible, and most importantly,
without errors.

That is, after this KIP, if the broker restarts, and the same generation
number is reused, it won't break current rebalance behavior. But it'll help
the consumer lead do the sticky assignments correctly.

Thank you.
Luke

On Fri, Dec 3, 2021 at 6:30 AM Colin McCabe  wrote:

> How are we going to avoid the situation where the broker restarts, and the
> same generation number is reused?
>
> best,
> Colin
>
> On Tue, Nov 30, 2021, at 16:36, Luke Chen wrote:
> > Hi all,
> >
> > I'd like to start the vote for KIP-792: Add "generation" field into
> > consumer protocol.
> >
> > The goal of this KIP is to allow the assignor/consumer coordinator to
> have
> > a way to identify the out-of-date members/assignments, to avoid rebalance
> > stuck issues in current protocol.
> >
> > Detailed description can be found here:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
>

Re: [VOTE] KIP-799: Align behaviour for producer callbacks with documented behaviour

[Luke Chen]

Hi Séamus,

Thanks for the update.
Looks better now!

Thank you.
Luke

On Sat, Dec 4, 2021 at 12:57 AM Séamus Ó Ceanainn <
seamus.oceana...@zalando.ie> wrote:

> Hey Luke,
>
> Thanks for the feedback. I've updated the relevant section to hopefully
> make it more clear from the KIP itself what placeholder value would be
> returned.
>
> Regards,
> Séamus.
>
> On Tue, 30 Nov 2021 at 09:52, Luke Chen  wrote:
>
> > Hi Séamus,
> > Thanks for the KIP!
> > We definitely want to keep the producer callback consistent for all types
> > of errors.
> >
> > Just one comment for the KIP:
> > In the "Proposed Changes" section, could you please "explicitly" describe
> > what placeholder you'll return, in addition to adding a hyperlink to
> other
> > places, to make it clear.
> >
> > +1 (non-binding)
> >
> > Thank you.
> > Luke
> >
> > On Tue, Nov 30, 2021 at 1:17 PM John Roesler 
> wrote:
> >
> > > Thanks, Séamus!
> > >
> > > I'm +1 (binding).
> > >
> > > On Mon, 2021-11-29 at 16:14 +, Séamus Ó Ceanainn wrote:
> > > > Hi everyone,
> > > >
> > > > I'd like to start a vote for KIP-799: Align behaviour for producer
> > > > callbacks with documented behaviour
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-799%3A+Align+behaviour+for+producer+callbacks+with+documented+behaviour
> > > >
> > > > .
> > > >
> > > > The KIP proposes a breaking change in the behaviour of producer
> client
> > > > callbacks. The breaking change would align the behaviour of callbacks
> > > with
> > > > the documented behaviour for the method, and makes it consistent with
> > > > similar methods for producer client interceptors.
> > > >
> > > > Regards,
> > > > Séamus.
> > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC member: David Jacot

[Luke Chen]

Congrats, David!
Well deserved.

Luke

deng ziming  於 2021年12月18日 週六 上午7:47 寫道：

> Congrats David!
>
> --
> Ziming Deng
>
> > On Dec 18, 2021, at 7:08 AM, Gwen Shapira  wrote:
> >
> > Hi everyone,
> >
> > David Jacot has been an Apache Kafka committer since Oct 2020 and has
> been contributing to the community consistently this entire time -
> especially notable the fact that he reviewed around 150 PRs in the last
> year. It is my pleasure to announce that David agreed to join the Kafka PMC.
> >
> > Congratulations, David!
> >
> > Gwen Shapira, on behalf of Apache Kafka PMC
>
>