Re: apply for permission to contribute to Apache Kafka

[Doe John]

Thanks,

After obtaining permission, I want to assign this JIRA ticket

to myself, but there's no 「Assign」 button for me.
[image: image.png]
Is there any problem here?

Best Regards,
John Doe



Luke Chen  于2023年5月10日周三 01:04写道：

> Done.
>
> Thanks.
> Luke
>
> On Sat, May 6, 2023 at 9:38 PM Doe John  wrote:
>
> > my Jira ID: johndoe
> >
> > on email zh2725284...@gmail.com
> >
> > Thanks!
> >
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Ismael Juma]

Hi Luke,

As discussed in the other KIP, there are some subtleties when it comes to
the semantics of the system if we don't wait for all members of the isr
before we ack. I don't understand why you say the leader election question
is out of scope - it seems to be a core aspect to me.

Ismael


On Wed, May 10, 2023, 8:50 AM Luke Chen  wrote:

> Hi Ismael,
>
> No, I didn't know about this similar KIP! I hope I've known that so that I
> don't need to spend time to write it again! :(
> I checked the KIP and all the discussions (here
> ). I
> think the consensus is that adding a client config to `acks=quorum` is
> fine.
> This comment
>  from
> Guozhang pretty much concluded what I'm trying to do.
>
>
>
>
>
>
>
>
> *1. Add one more value to client-side acks config:   0: no acks needed at
> all.   1: ack from the leader.   all: ack from ALL the ISR replicas
>  quorum: this is the new value, it requires ack from enough number of ISR
> replicas no smaller than majority of the replicas AND no smaller
> than{min.isr}.2. Clarify in the docs that if a user wants to tolerate X
> failures, she needs to set client acks=all or acks=quorum (better tail
> latency than "all") with broker {min.sir} to be X+1; however, "all" is not
> necessarily stronger than "quorum".*
>
> Concerns from KIP-250 are:
> 1. Introducing a new leader LEO based election method. This is not clear in
> the KIP-250 and needs more discussion
> 2. The KIP-250 also tried to optimize the consumer latency to read messages
> beyond high watermark, which also has some discussion about how to achieve
> that, and no conclusion
>
> Both of the above 2 concerns are out of the scope of my current KIP.
> So, I think it's good to provide this `acks=quorum` or
> `acks=min.insync.replicas` option to users to give them another choice.
>
>
> Thank you.
> Luke
>
>
> On Wed, May 10, 2023 at 8:54 AM Ismael Juma  wrote:
>
> > Hi Luke,
> >
> > Are you aware of
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-250+Add+Support+for+Quorum-based+Producer+Acknowledgment
> > ?
> >
> > Ismael
> >
> > On Tue, May 9, 2023 at 10:14 PM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion for the KIP-926: introducing
> > > acks=min.insync.replicas config. This KIP is to introduce
> > > `acks=min.insync.replicas` config value in producer, to improve the
> write
> > > throughput and still guarantee high durability.
> > >
> > > Please check the link for more detail:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> > >
> > > Any feedback is welcome.
> > >
> > > Thank you.
> > > Luke
> > >
> >
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Ismael,

No, I didn't know about this similar KIP! I hope I've known that so that I
don't need to spend time to write it again! :(
I checked the KIP and all the discussions (here
). I
think the consensus is that adding a client config to `acks=quorum` is
fine.
This comment
 from
Guozhang pretty much concluded what I'm trying to do.








*1. Add one more value to client-side acks config:   0: no acks needed at
all.   1: ack from the leader.   all: ack from ALL the ISR replicas
 quorum: this is the new value, it requires ack from enough number of ISR
replicas no smaller than majority of the replicas AND no smaller
than{min.isr}.2. Clarify in the docs that if a user wants to tolerate X
failures, she needs to set client acks=all or acks=quorum (better tail
latency than "all") with broker {min.sir} to be X+1; however, "all" is not
necessarily stronger than "quorum".*

Concerns from KIP-250 are:
1. Introducing a new leader LEO based election method. This is not clear in
the KIP-250 and needs more discussion
2. The KIP-250 also tried to optimize the consumer latency to read messages
beyond high watermark, which also has some discussion about how to achieve
that, and no conclusion

Both of the above 2 concerns are out of the scope of my current KIP.
So, I think it's good to provide this `acks=quorum` or
`acks=min.insync.replicas` option to users to give them another choice.


Thank you.
Luke


On Wed, May 10, 2023 at 8:54 AM Ismael Juma  wrote:

> Hi Luke,
>
> Are you aware of
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-250+Add+Support+for+Quorum-based+Producer+Acknowledgment
> ?
>
> Ismael
>
> On Tue, May 9, 2023 at 10:14 PM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-926: introducing
> > acks=min.insync.replicas config. This KIP is to introduce
> > `acks=min.insync.replicas` config value in producer, to improve the write
> > throughput and still guarantee high durability.
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>

Re: [DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

[Colt McNealy]

Hello Hao,

First of all, THANK YOU for putting this together. I had been hoping
someone might bring something like this forward. A few comments:

**1: Runtime Complexity
> Klein’s cycle canceling algorithm can solve the min-cost flow problem in
O(E^2CU) time where C is max cost and U is max capacity. In our particular
case, C is 1 and U is at most 3 (A task can have at most 3 topics including
changelog topic?). So the algorithm runs in O(E^2) time for our case.

A Task can have multiple input topics, and also multiple state stores, and
multiple output topics. The most common case is three topics as you
described, but this is not necessarily guaranteed. Also, math is one of my
weak points, but to me O(E^2) is equivalent to O(1), and I struggle to see
how the algorithm isn't at least O(N) where N is the number of Tasks...?

**2: Broker-Side Partition Assignments
Consider the case with just three topics in a Task (one input, one output,
one changelog). If all three partition leaders are in the same Rack (or
better yet, the same broker), then we could get massive savings by
assigning the Task to that Rack/availability zone. But if the leaders for
each partition are spread across multiple zones, how will you handle that?
Is that outside the scope of this KIP, or is it worth introducing a
kafka-streams-generate-rebalance-proposal.sh tool?

Colt McNealy
*Founder, LittleHorse.io*


On Tue, May 9, 2023 at 4:03 PM Hao Li  wrote:

> Hi all,
>
> I have submitted KIP-925 to add rack awareness logic in task assignment in
> Kafka Streams and would like to start a discussion:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
>
> --
> Thanks,
> Hao
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Ismael Juma]

Hi Luke,

Are you aware of
https://cwiki.apache.org/confluence/display/KAFKA/KIP-250+Add+Support+for+Quorum-based+Producer+Acknowledgment
?

Ismael

On Tue, May 9, 2023 at 10:14 PM Luke Chen  wrote:

> Hi all,
>
> I'd like to start a discussion for the KIP-926: introducing
> acks=min.insync.replicas config. This KIP is to introduce
> `acks=min.insync.replicas` config value in producer, to improve the write
> throughput and still guarantee high durability.
>
> Please check the link for more detail:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
>
> Any feedback is welcome.
>
> Thank you.
> Luke
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #175

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Justine,

Thanks for the comment. Yes, I know we have a configuration
min.insync.replicas to avoid this case. And actually, that's my following
paragraph is talking about.

Please let me know if you have other comments.

Thanks
Luke

Justine Olshan  於 2023年5月10日 週三 上午1:01 寫道：

> Hey Luke,
>
> I was taking a quick pass over the KIP and saw this line:
>
> >It looks perfect. But there's a caveat here. Like the doc said, acks=all
> will "wait for the *full set of in-sync replicas *to acknowledge the
> record", so if there's only 1 replica in in-sync replicas, it will have the
> same effect as acks=1 (even though we have replication-factor set to 3).
>
> I thought we had a configuration min.insync.replicas to avoid this case.
> (Typically it is set to two so we need a leader and one follower to
> respond.)
>
> I am curious to understand the proposal more, but just thought I'd share
> this as it came up.
>
> Justine
>
> On Tue, May 9, 2023 at 9:44 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-926: introducing
> > acks=min.insync.replicas config. This KIP is to introduce
> > `acks=min.insync.replicas` config value in producer, to improve the write
> > throughput and still guarantee high durability.
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>

[jira] [Created] (KAFKA-14981) Set `group.instance.id` in streams consumer so that rebalance will not happen if a instance is restarted

[Hao Li (Jira)]

Hao Li created KAFKA-14981:
--

 Summary: Set `group.instance.id` in streams consumer so that 
rebalance will not happen if a instance is restarted
 Key: KAFKA-14981
 URL: https://issues.apache.org/jira/browse/KAFKA-14981
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Hao Li


`group.instance.id` enables static membership so that if a consumer is 
restarted within `session.timeout.ms`, rebalance will not be triggered and 
originally assignment can be returned directly from broker. We can set this id 
in Kafka streams using `threadId` so that no rebalance is trigger within 
`session.timeout.ms`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-925: rack aware task assignment in Kafka Streams

[Hao Li]

Hi all,

I have submitted KIP-925 to add rack awareness logic in task assignment in
Kafka Streams and would like to start a discussion:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams

-- 
Thanks,
Hao

Call for Presentations, Community Over Code 2023

[Rich Bowen]

(Note: You are receiving this because you are subscribed to the dev@
list for one or more Apache Software Foundation projects.)

The Call for Presentations (CFP) for Community Over Code (formerly
Apachecon) 2023 is open at
https://communityovercode.org/call-for-presentations/, and will close
Thu, 13 Jul 2023 23:59:59 GMT.

The event will be held in Halifax, Canada, October 7-10, 2023.

We welcome submissions on any topic related to the Apache Software
Foundation, Apache projects, or the communities around those projects.
We are specifically looking for presentations in the following
catetegories:

Fintech
Search
Big Data, Storage
Big Data, Compute
Internet of Things
Groovy
Incubator
Community
Data Engineering
Performance Engineering
Geospatial
API/Microservices
Frameworks
Content Wrangling
Tomcat and httpd
Cloud and Runtime
Streaming
Sustainability

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Sophie Blee-Goldman]

Hey Mickael, I noticed a bug in the new versioned key-value byte store
where it's delegating to the wrong API
(copy/paste error I assume). I extracted this into its own PR which I think
should be included in the 3.5 release.

The tests are still running, but it's just a one-liner so I'll merge it
when they're done, and cherrypick to 3.5 if
that's ok with you. See https://github.com/apache/kafka/pull/13695

Thanks for running the release!

On Tue, May 9, 2023 at 1:28 PM Randall Hauch  wrote:

> Thanks, Mickael.
>
> I've cherry-picked that commit to the `3.5` branch (
> https://issues.apache.org/jira/browse/KAFKA-14974).
>
> Best regards,
> Randall
>
> On Tue, May 9, 2023 at 2:13 PM Mickael Maison 
> wrote:
>
> > Hi Randall/Luke,
> >
> > Yes you can go ahead and merge these into 3.5. I've not started making
> > a release yet because:
> > - I found a regression today in MirrorMaker:
> > https://issues.apache.org/jira/browse/KAFKA-14980
> > - The 3.5 branch builder job in Jenkins has been disabled:
> > https://issues.apache.org/jira/browse/INFRA-24577
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 9, 2023 at 8:40 PM Luke Chen  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Since we haven't had the CR created yet, I'm thinking we should
> backport
> > > this doc improvement to v3.5.0 to make the doc complete.
> > > https://github.com/apache/kafka/pull/13660
> > >
> > > What do you think?
> > >
> > > Luke
> > >
> > > On Sat, May 6, 2023 at 11:31 PM David Arthur  wrote:
> > >
> > > > I resolved these three:
> > > > * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix
> > version
> > > > * KAFKA-14805 is merged to trunk and 3.5
> > > > * KAFKA-14918 is merged to trunk and 3.5
> > > >
> > > > KAFKA-14692 (docs issue) is still a not done
> > > >
> > > > Looks like KAFKA-14084 is now resolved as well (it's in trunk and
> 3.5).
> > > >
> > > > I'll try to find out about KAFKA-14698, I think it's likely a
> WONTFIX.
> > > >
> > > > -David
> > > >
> > > > On Fri, May 5, 2023 at 10:43 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > Thanks for the update!
> > > > > You still own 4 other tickets targeting 3.5: KAFKA-14840,
> > KAFKA-14805,
> > > > > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > > > > release?
> > > > > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > > > > migration. Should I move them too?
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Fri, May 5, 2023 at 4:27 PM David Arthur
> > > > >  wrote:
> > > > > >
> > > > > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > > > > >
> > > > > > Cheers,
> > > > > > David
> > > > > >
> > > > > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Divij,
> > > > > > >
> > > > > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have
> > been
> > > > > > > updated since 3.4.
> > > > > > > Regarding your PR, I would have been in favor of bringing this
> > to 3.5
> > > > > > > a couple of weeks ago, but we're now a week past code freeze
> for
> > 3.5.
> > > > > > > Apart if this fixes CVEs, or significant bugs, I think we
> should
> > only
> > > > > > > merge it in trunk.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya <
> > divijvaidy...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hey Mickael
> > > > > > > >
> > > > > > > > Should we consider performing an update of the minor versions
> > of
> > > > the
> > > > > > > > dependencies in 3.5 (per
> > > > https://github.com/apache/kafka/pull/13673
> > > > > )?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Divij Vaidya
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Luke,
> > > > > > > > >
> > > > > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen <
> show...@gmail.com
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mickael,
> > > > > > > > > >
> > > > > > > > > > There are 1 bug and 1 improvement that I'd like to
> > backport to
> > > > > 3.5.
> > > > > > > > > > 1. A small improvement for ZK migration based on
> > KAFKA-14840
> > > > > > > (mentioned
> > > > > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > > > > >
> > > > > > > > > > 2. A bug will cause the KRaft controller node to shut
> down
> > > > > > > unexpectedly.
> > > > > > > > > PR
> > > > > > > > > > is ready for review.
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #174

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Randall Hauch]

Thanks, Mickael.

I've cherry-picked that commit to the `3.5` branch (
https://issues.apache.org/jira/browse/KAFKA-14974).

Best regards,
Randall

On Tue, May 9, 2023 at 2:13 PM Mickael Maison 
wrote:

> Hi Randall/Luke,
>
> Yes you can go ahead and merge these into 3.5. I've not started making
> a release yet because:
> - I found a regression today in MirrorMaker:
> https://issues.apache.org/jira/browse/KAFKA-14980
> - The 3.5 branch builder job in Jenkins has been disabled:
> https://issues.apache.org/jira/browse/INFRA-24577
>
> Thanks,
> Mickael
>
> On Tue, May 9, 2023 at 8:40 PM Luke Chen  wrote:
> >
> > Hi Mickael,
> >
> > Since we haven't had the CR created yet, I'm thinking we should backport
> > this doc improvement to v3.5.0 to make the doc complete.
> > https://github.com/apache/kafka/pull/13660
> >
> > What do you think?
> >
> > Luke
> >
> > On Sat, May 6, 2023 at 11:31 PM David Arthur  wrote:
> >
> > > I resolved these three:
> > > * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix
> version
> > > * KAFKA-14805 is merged to trunk and 3.5
> > > * KAFKA-14918 is merged to trunk and 3.5
> > >
> > > KAFKA-14692 (docs issue) is still a not done
> > >
> > > Looks like KAFKA-14084 is now resolved as well (it's in trunk and 3.5).
> > >
> > > I'll try to find out about KAFKA-14698, I think it's likely a WONTFIX.
> > >
> > > -David
> > >
> > > On Fri, May 5, 2023 at 10:43 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Thanks for the update!
> > > > You still own 4 other tickets targeting 3.5: KAFKA-14840,
> KAFKA-14805,
> > > > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > > > release?
> > > > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > > > migration. Should I move them too?
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, May 5, 2023 at 4:27 PM David Arthur
> > > >  wrote:
> > > > >
> > > > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > > > >
> > > > > Cheers,
> > > > > David
> > > > >
> > > > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Divij,
> > > > > >
> > > > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have
> been
> > > > > > updated since 3.4.
> > > > > > Regarding your PR, I would have been in favor of bringing this
> to 3.5
> > > > > > a couple of weeks ago, but we're now a week past code freeze for
> 3.5.
> > > > > > Apart if this fixes CVEs, or significant bugs, I think we should
> only
> > > > > > merge it in trunk.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya <
> divijvaidy...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hey Mickael
> > > > > > >
> > > > > > > Should we consider performing an update of the minor versions
> of
> > > the
> > > > > > > dependencies in 3.5 (per
> > > https://github.com/apache/kafka/pull/13673
> > > > )?
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Luke,
> > > > > > > >
> > > > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen  >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Mickael,
> > > > > > > > >
> > > > > > > > > There are 1 bug and 1 improvement that I'd like to
> backport to
> > > > 3.5.
> > > > > > > > > 1. A small improvement for ZK migration based on
> KAFKA-14840
> > > > > > (mentioned
> > > > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > > > >
> > > > > > > > > 2. A bug will cause the KRaft controller node to shut down
> > > > > > unexpectedly.
> > > > > > > > PR
> > > > > > > > > is ready for review.
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > > > > > https://github.com/apache/kafka/pull/13653
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi David,
> > > > > > > > > >
> > > > > > > > > > Yes you can backport these to 3.5. Let me know when you
> are
> > > > done.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > > > > > > >  wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hey Mickael,
> > > > > > > > > > >
> > > > > > > > > > > I have one major ZK migration 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1832

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi Randall/Luke,

Yes you can go ahead and merge these into 3.5. I've not started making
a release yet because:
- I found a regression today in MirrorMaker:
https://issues.apache.org/jira/browse/KAFKA-14980
- The 3.5 branch builder job in Jenkins has been disabled:
https://issues.apache.org/jira/browse/INFRA-24577

Thanks,
Mickael

On Tue, May 9, 2023 at 8:40 PM Luke Chen  wrote:
>
> Hi Mickael,
>
> Since we haven't had the CR created yet, I'm thinking we should backport
> this doc improvement to v3.5.0 to make the doc complete.
> https://github.com/apache/kafka/pull/13660
>
> What do you think?
>
> Luke
>
> On Sat, May 6, 2023 at 11:31 PM David Arthur  wrote:
>
> > I resolved these three:
> > * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix version
> > * KAFKA-14805 is merged to trunk and 3.5
> > * KAFKA-14918 is merged to trunk and 3.5
> >
> > KAFKA-14692 (docs issue) is still a not done
> >
> > Looks like KAFKA-14084 is now resolved as well (it's in trunk and 3.5).
> >
> > I'll try to find out about KAFKA-14698, I think it's likely a WONTFIX.
> >
> > -David
> >
> > On Fri, May 5, 2023 at 10:43 AM Mickael Maison 
> > wrote:
> >
> > > Hi David,
> > >
> > > Thanks for the update!
> > > You still own 4 other tickets targeting 3.5: KAFKA-14840, KAFKA-14805,
> > > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > > release?
> > > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > > migration. Should I move them too?
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Fri, May 5, 2023 at 4:27 PM David Arthur
> > >  wrote:
> > > >
> > > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > > >
> > > > Cheers,
> > > > David
> > > >
> > > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Divij,
> > > > >
> > > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> > > > > updated since 3.4.
> > > > > Regarding your PR, I would have been in favor of bringing this to 3.5
> > > > > a couple of weeks ago, but we're now a week past code freeze for 3.5.
> > > > > Apart if this fixes CVEs, or significant bugs, I think we should only
> > > > > merge it in trunk.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya  > >
> > > > > wrote:
> > > > > >
> > > > > > Hey Mickael
> > > > > >
> > > > > > Should we consider performing an update of the minor versions of
> > the
> > > > > > dependencies in 3.5 (per
> > https://github.com/apache/kafka/pull/13673
> > > )?
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Luke,
> > > > > > >
> > > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen 
> > > wrote:
> > > > > > > >
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > There are 1 bug and 1 improvement that I'd like to backport to
> > > 3.5.
> > > > > > > > 1. A small improvement for ZK migration based on KAFKA-14840
> > > > > (mentioned
> > > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > > >
> > > > > > > > 2. A bug will cause the KRaft controller node to shut down
> > > > > unexpectedly.
> > > > > > > PR
> > > > > > > > is ready for review.
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > > > > https://github.com/apache/kafka/pull/13653
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Luke
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi David,
> > > > > > > > >
> > > > > > > > > Yes you can backport these to 3.5. Let me know when you are
> > > done.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > Hey Mickael,
> > > > > > > > > >
> > > > > > > > > > I have one major ZK migration improvement (KAFKA-14805)
> > that
> > > > > landed
> > > > > > > in
> > > > > > > > > > trunk this week that I'd like to merge to 3.5 (once we fix
> > > some
> > > > > test
> > > > > > > > > > failures it introduced). After that, I have another PR for
> > > > > > > KAFKA-14840
> > > > > > > > > > which is essentially a huge bug in the ZK migration logic
> > > that
> > > > > needs
> > > > > > > to
> > > > > > > > > > land in 3.5.
> > > > > > > > > >
> > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > > > > > > > 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Luke Chen]

Hi Mickael,

Since we haven't had the CR created yet, I'm thinking we should backport
this doc improvement to v3.5.0 to make the doc complete.
https://github.com/apache/kafka/pull/13660

What do you think?

Luke

On Sat, May 6, 2023 at 11:31 PM David Arthur  wrote:

> I resolved these three:
> * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix version
> * KAFKA-14805 is merged to trunk and 3.5
> * KAFKA-14918 is merged to trunk and 3.5
>
> KAFKA-14692 (docs issue) is still a not done
>
> Looks like KAFKA-14084 is now resolved as well (it's in trunk and 3.5).
>
> I'll try to find out about KAFKA-14698, I think it's likely a WONTFIX.
>
> -David
>
> On Fri, May 5, 2023 at 10:43 AM Mickael Maison 
> wrote:
>
> > Hi David,
> >
> > Thanks for the update!
> > You still own 4 other tickets targeting 3.5: KAFKA-14840, KAFKA-14805,
> > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > release?
> > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > migration. Should I move them too?
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 5, 2023 at 4:27 PM David Arthur
> >  wrote:
> > >
> > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > >
> > > Cheers,
> > > David
> > >
> > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Divij,
> > > >
> > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> > > > updated since 3.4.
> > > > Regarding your PR, I would have been in favor of bringing this to 3.5
> > > > a couple of weeks ago, but we're now a week past code freeze for 3.5.
> > > > Apart if this fixes CVEs, or significant bugs, I think we should only
> > > > merge it in trunk.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya  >
> > > > wrote:
> > > > >
> > > > > Hey Mickael
> > > > >
> > > > > Should we consider performing an update of the minor versions of
> the
> > > > > dependencies in 3.5 (per
> https://github.com/apache/kafka/pull/13673
> > )?
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen 
> > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > There are 1 bug and 1 improvement that I'd like to backport to
> > 3.5.
> > > > > > > 1. A small improvement for ZK migration based on KAFKA-14840
> > > > (mentioned
> > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > >
> > > > > > > 2. A bug will cause the KRaft controller node to shut down
> > > > unexpectedly.
> > > > > > PR
> > > > > > > is ready for review.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > > > https://github.com/apache/kafka/pull/13653
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > Yes you can backport these to 3.5. Let me know when you are
> > done.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hey Mickael,
> > > > > > > > >
> > > > > > > > > I have one major ZK migration improvement (KAFKA-14805)
> that
> > > > landed
> > > > > > in
> > > > > > > > > trunk this week that I'd like to merge to 3.5 (once we fix
> > some
> > > > test
> > > > > > > > > failures it introduced). After that, I have another PR for
> > > > > > KAFKA-14840
> > > > > > > > > which is essentially a huge bug in the ZK migration logic
> > that
> > > > needs
> > > > > > to
> > > > > > > > > land in 3.5.
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14805 (done)
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14840 (nearly
> > done)
> > > > > > > > >
> > > > > > > > > I just wanted to check with you before cherry-picking these
> > to
> > > > 3.5
> > > > > > > > >
> > > > > > > > > David
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Apr 24, 2023 at 1:18 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Justine,
> > > > > > > > > >
> > > > > > > > > > That makes sense. Feel free to revert that commit in 3.5.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mickael
> > > > > > > > > >
> > > > > > > > > > On Mon, Apr 24, 2023 at 7:16 PM Mickael 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1828

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 566880 lines...]
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 11 and Scala 2.13
[2023-05-08T18:03:54.224Z] 
[2023-05-08T18:03:54.224Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 136 > 
org.apache.kafka.connect.integration.TransformationIntegrationTest > 
testFilterOnTombstonesWithSinkConnector PASSED
[2023-05-08T18:03:54.224Z] 
[2023-05-08T18:03:54.224Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 136 > 
org.apache.kafka.connect.integration.TransformationIntegrationTest > 
testFilterOnTopicNameWithSinkConnector STARTED
[2023-05-08T18:04:12.357Z] 
[2023-05-08T18:04:12.357Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 136 > 
org.apache.kafka.connect.integration.TransformationIntegrationTest > 
testFilterOnTopicNameWithSinkConnector PASSED
[2023-05-08T18:04:18.896Z] 
[2023-05-08T18:04:18.896Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 186 > 
org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testTaskStatuses STARTED
[2023-05-08T18:04:22.174Z] 
[2023-05-08T18:04:22.174Z] > Task :connect:mirror:integrationTest
[2023-05-08T18:04:22.174Z] 
[2023-05-08T18:04:22.174Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 131 > MirrorConnectorsIntegrationSSLTest > 
testNoCheckpointsIfNoRecordsAreMirrored() PASSED
[2023-05-08T18:04:22.174Z] 
[2023-05-08T18:04:22.174Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 131 > MirrorConnectorsIntegrationSSLTest > 
testOneWayReplicationWithAutoOffsetSync() STARTED
[2023-05-08T18:04:33.328Z] 
[2023-05-08T18:04:33.328Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 130 > MirrorConnectorsIntegrationExactlyOnceTest > 
testOffsetTranslationBehindReplicationFlow() PASSED
[2023-05-08T18:04:33.328Z] 
[2023-05-08T18:04:33.328Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 130 > MirrorConnectorsIntegrationExactlyOnceTest > 
testNoCheckpointsIfNoRecordsAreMirrored() STARTED
[2023-05-08T18:04:54.690Z] 
[2023-05-08T18:04:54.690Z] > Task :connect:runtime:integrationTest
[2023-05-08T18:04:54.690Z] 
[2023-05-08T18:04:54.690Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 186 > 
org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testTaskStatuses PASSED
[2023-05-08T18:04:54.690Z] 
[2023-05-08T18:04:54.690Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 186 > 
org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testPauseStopResume STARTED
[2023-05-08T18:05:07.838Z] 
[2023-05-08T18:05:07.838Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 186 > 
org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest > 
testPauseStopResume PASSED
[2023-05-08T18:05:17.243Z] WARNING: An illegal reflective access operation has 
occurred
[2023-05-08T18:05:17.243Z] WARNING: Illegal reflective access by 
org.powermock.reflect.internal.WhiteboxImpl 
(file:/home/jenkins/.gradle/caches/modules-2/files-2.1/org.powermock/powermock-reflect/2.0.9/4bb9ed43e5221926fb86cae44b445de110a51d05/powermock-reflect-2.0.9.jar)
 to method java.lang.String.value()
[2023-05-08T18:05:17.243Z] WARNING: Please consider reporting this to the 
maintainers of org.powermock.reflect.internal.WhiteboxImpl
[2023-05-08T18:05:17.243Z] WARNING: Use --illegal-access=warn to enable 
warnings of further illegal reflective access operations
[2023-05-08T18:05:17.243Z] WARNING: All illegal access operations will be 
denied in a future release
[2023-05-08T18:05:32.705Z] 
[2023-05-08T18:05:32.705Z] 92 tests completed, 2 failed, 1 skipped
[2023-05-08T18:05:38.039Z] There were failing tests. See the report at: 
file:///home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka_trunk@2/connect/runtime/build/reports/tests/integrationTest/index.html
[2023-05-08T18:05:47.442Z] 
[2023-05-08T18:05:47.442Z] > Task :connect:mirror:integrationTest
[2023-05-08T18:05:47.442Z] 
[2023-05-08T18:05:47.442Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 130 > MirrorConnectorsIntegrationExactlyOnceTest > 
testNoCheckpointsIfNoRecordsAreMirrored() PASSED
[2023-05-08T18:05:47.442Z] 
[2023-05-08T18:05:47.442Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 130 > MirrorConnectorsIntegrationExactlyOnceTest > 
testOneWayReplicationWithAutoOffsetSync() STARTED
[2023-05-08T18:05:52.771Z] 
[2023-05-08T18:05:52.772Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 131 > MirrorConnectorsIntegrationSSLTest > 
testOneWayReplicationWithAutoOffsetSync() PASSED
[2023-05-08T18:05:52.772Z] 
[2023-05-08T18:05:52.772Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 131 > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1830

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1829

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1825

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 470690 lines...]
[2023-05-08T05:30:27.978Z] 
[2023-05-08T05:30:27.978Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > 
shouldSetChangelogTopicProperties PASSED
[2023-05-08T05:30:27.978Z] 
[2023-05-08T05:30:27.978Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > shouldRestore 
STARTED
[2023-05-08T05:31:18.065Z] 
[2023-05-08T05:31:18.065Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > shouldRestore PASSED
[2023-05-08T05:31:18.065Z] 
[2023-05-08T05:31:18.065Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > 
shouldPutGetAndDelete STARTED
[2023-05-08T05:31:18.065Z] 
[2023-05-08T05:31:18.065Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > 
shouldPutGetAndDelete PASSED
[2023-05-08T05:31:18.065Z] 
[2023-05-08T05:31:18.065Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned STARTED
[2023-05-08T05:32:09.403Z] 
[2023-05-08T05:32:09.403Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > VersionedKeyValueStoreIntegrationTest > 
shouldManualUpgradeFromNonVersionedTimestampedToVersioned PASSED
[2023-05-08T05:32:09.403Z] 
[2023-05-08T05:32:09.403Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted STARTED
[2023-05-08T05:32:15.368Z] 
[2023-05-08T05:32:15.368Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > HandlingSourceTopicDeletionIntegrationTest > 
shouldThrowErrorAfterSourceTopicDeleted PASSED
[2023-05-08T05:32:16.390Z] OpenJDK 64-Bit Server VM warning: Sharing is only 
supported for boot loader classes because bootstrap classpath has been appended
[2023-05-08T05:32:16.390Z] 
[2023-05-08T05:32:16.390Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers STARTED
[2023-05-08T05:32:41.013Z] 
[2023-05-08T05:32:41.013Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2023-05-08T05:32:41.013Z] 
[2023-05-08T05:32:41.013Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2023-05-08T05:32:54.306Z] 
[2023-05-08T05:32:54.306Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2023-05-08T05:32:54.306Z] 
[2023-05-08T05:32:54.306Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2023-05-08T05:32:54.306Z] 
[2023-05-08T05:32:54.306Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient PASSED
[2023-05-08T05:32:54.306Z] 
[2023-05-08T05:32:54.306Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys STARTED
[2023-05-08T05:33:01.173Z] 
[2023-05-08T05:33:01.173Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyStandbys PASSED
[2023-05-08T05:33:01.173Z] 
[2023-05-08T05:33:01.173Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient STARTED
[2023-05-08T05:33:02.427Z] 
[2023-05-08T05:33:02.427Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testStickyTaskAssignorManyThreadsPerClient PASSED
[2023-05-08T05:33:02.427Z] 
[2023-05-08T05:33:02.427Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient STARTED
[2023-05-08T05:33:04.377Z] 
[2023-05-08T05:33:04.377Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyThreadsPerClient PASSED
[2023-05-08T05:33:04.377Z] 
[2023-05-08T05:33:04.377Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargePartitionCount STARTED
[2023-05-08T05:33:37.091Z] 

Re: KafkaConsumer refactor proposal

[Philip Nee]

Hey Divij,

Firstly, THANK YOU for reaching out and providing the feedback.  At this
moment, I don't have all of the answers to your questions at the moment,
but I hope my responses could be helpful in moving the project forward.

1. There isn't a prioritization strategy, and the network request buffer is
just FIFO. Basically, all requests are buffered in the network queue and
sent out at the end of the event loop where we poll the client.
Presumably, your concern is that the member can get constantly kicked out
of the group due to not sending out a heartbeat soon enough under high load
or network congestion.  I think that's a fair point.  On the other hand, we
design the background thread (or we would like the design to be...) to be
lightweight and focus on network communication, such that the client can
quickly go through all the requests without consuming too many resources.
In the case of network congestion, I think there could be other reasons
that cause members to fail, so I don't know if we need the prioritize the
requests.

2. Yes, as of now, the fetch offset request is being de-duped, and we would
do that for other requests as well.  There's also a RequestState class that
helps request managers track retry backoff (exponential backoff), which
prevents members from spamming brokers.

3. There's no implementation of the backpressure for both the application
and background event queue, but here is my thought:
 - For ApplicationEventQueue: I can think of a few ways to implement this.
1. user handles a RetriableError and re-enqueues the event. or 2. Having
the application to buffer the events, but then we are just implementing a
queue on top of the queue.  What's your suggestion here?
 - For BackgroundEventQueue: The main events are fetches, errors, and
rebalance triggers.  For fetching, what's in my mind is some kind of memory
bound mechanism like KIP-81

(it
hasn't been implemented).  For rebalancing events or errors, there is not
much we could do aside from pausing the request managers.  What do you
think?

4. See above.

5. I'll add an Observability section to the one-pager. My thinking is
similar to yours: 1. memory and CPU pressure and 2. event latency.

I would love to incorporate some of your past experience from TinkerPop
into this design, so thank you again for sharing your feedback!

Thank you!
P

On Tue, May 9, 2023 at 9:50 AM Divij Vaidya  wrote:

> Hey Philip
>
> Thank you for writing this down.
>
> Overall, the ideal of decoupling the responsibility amongst threads and
> communication using queues looks good. I have a few questions
> associated with handling failure scenarios.
>
> 1. I observed that there is no concept of prioritization in the background
> thread. In case of high load, should a heartbeat handler be prioritized
> over listOffset handler?
> 2. Idempotent requests could be "bulked" together, e.g. if we have two list
> offsets in the queue, we should be able to call listOffset once and
> duplicate the response for the requests
> 3. What would the back pressure handling behaviour from the
> applicationEventQueue look like? This is bound to occur during intermittent
> network disruption
> 4. Same as 3 but for backgroundEventQueue. If the client thread is doing
> some I/O since the user is blocking it in their business logic, then the
> queue may get full. What happens then? How do we recover and cancel the
> resources such as waiting sync APIs?
> 5. Can we add observability/metrics in the design? As an example, the end
> user might measure latency from the client thread but the request may
> actually be spending time in the queue waiting to be picked by the
> background thread. In such cases, user should have the visibility to
> identify that a need to scale the background thread processing is required.
>
> --
> Divij Vaidya
>
>
>
> On Sun, Sep 18, 2022 at 11:24 PM Philip Nee  wrote:
>
> > On Sun, Sep 18, 2022 at 6:03 AM Luke Chen  wrote:
> >
> > > Hi Philip,
> > >
> > > Thanks for the write-up.
> > >
> > Also thank you for taking the time to read the proposal.  Very grateful.
> >
> > > Some questions:
> > >
> > > 1. You said: If we don't need a coordinator, the background thread will
> > > stay in the *initialized* state.
> > > But in the definition of *initialized, *it said:
> > > *initialized*: The background thread completes initialization, and the
> > loop
> > > has started.
> > > Does that mean, even if we don't need a coordinator, we still create a
> > > coordinator and run the loop? And why?
> > >
> > If we don't need a coordinator, I think the background thread should not
> > fire a FindCoordinator request, which is what is implemented today right?
> > In terms of object instantiation, I think we can instantiate the object
> on
> > demand.
> >
> > >
> > > 2. Why do we need so many rebalance states?
> > > After this proposal, we'll change the state number from 4 to 8.

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #124

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1826

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #125

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1831

[Apache Jenkins Server]

See 

Re: apply for permission to contribute to Apache Kafka

[Luke Chen]

Done.

Thanks.
Luke

On Sat, May 6, 2023 at 9:38 PM Doe John  wrote:

> my Jira ID: johndoe
>
> on email zh2725284...@gmail.com
>
> Thanks!
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Justine Olshan]

Hey Luke,

I was taking a quick pass over the KIP and saw this line:

>It looks perfect. But there's a caveat here. Like the doc said, acks=all
will "wait for the *full set of in-sync replicas *to acknowledge the
record", so if there's only 1 replica in in-sync replicas, it will have the
same effect as acks=1 (even though we have replication-factor set to 3).

I thought we had a configuration min.insync.replicas to avoid this case.
(Typically it is set to two so we need a leader and one follower to
respond.)

I am curious to understand the proposal more, but just thought I'd share
this as it came up.

Justine

On Tue, May 9, 2023 at 9:44 AM Luke Chen  wrote:

> Hi all,
>
> I'd like to start a discussion for the KIP-926: introducing
> acks=min.insync.replicas config. This KIP is to introduce
> `acks=min.insync.replicas` config value in producer, to improve the write
> throughput and still guarantee high durability.
>
> Please check the link for more detail:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
>
> Any feedback is welcome.
>
> Thank you.
> Luke
>

Re: KafkaConsumer refactor proposal

[Divij Vaidya]

Hey Philip

Thank you for writing this down.

Overall, the ideal of decoupling the responsibility amongst threads and
communication using queues looks good. I have a few questions
associated with handling failure scenarios.

1. I observed that there is no concept of prioritization in the background
thread. In case of high load, should a heartbeat handler be prioritized
over listOffset handler?
2. Idempotent requests could be "bulked" together, e.g. if we have two list
offsets in the queue, we should be able to call listOffset once and
duplicate the response for the requests
3. What would the back pressure handling behaviour from the
applicationEventQueue look like? This is bound to occur during intermittent
network disruption
4. Same as 3 but for backgroundEventQueue. If the client thread is doing
some I/O since the user is blocking it in their business logic, then the
queue may get full. What happens then? How do we recover and cancel the
resources such as waiting sync APIs?
5. Can we add observability/metrics in the design? As an example, the end
user might measure latency from the client thread but the request may
actually be spending time in the queue waiting to be picked by the
background thread. In such cases, user should have the visibility to
identify that a need to scale the background thread processing is required.

--
Divij Vaidya



On Sun, Sep 18, 2022 at 11:24 PM Philip Nee  wrote:

> On Sun, Sep 18, 2022 at 6:03 AM Luke Chen  wrote:
>
> > Hi Philip,
> >
> > Thanks for the write-up.
> >
> Also thank you for taking the time to read the proposal.  Very grateful.
>
> > Some questions:
> >
> > 1. You said: If we don't need a coordinator, the background thread will
> > stay in the *initialized* state.
> > But in the definition of *initialized, *it said:
> > *initialized*: The background thread completes initialization, and the
> loop
> > has started.
> > Does that mean, even if we don't need a coordinator, we still create a
> > coordinator and run the loop? And why?
> >
> If we don't need a coordinator, I think the background thread should not
> fire a FindCoordinator request, which is what is implemented today right?
> In terms of object instantiation, I think we can instantiate the object on
> demand.
>
> >
> > 2. Why do we need so many rebalance states?
> > After this proposal, we'll change the state number from 4 to 8.
> > And we will have many state changes and state check in the code. Could
> you
> > explain why those states are necessary?
> > I can imagine, like the method: `rebalanceInProgress`, will need to check
> > if the current state in
> >
> >
> PREPARE_REVOCATION/REVOKING_PARTITION/PARTITION_REVOKED/PREPARING_REBALANCE/.../...
> > .
> > So, maybe you should explain why these states are necessary. To me, like
> > PREPARE_REVOCATION/REVOKING_PARTITION, I don't understand why we need 2
> > states for them? Any reason there?
> >
> I think we need to do 3 things when revoking a partition: There are, before
> the revocation, during the revocation, and after the revocation.
>   1. Before: prepare the revocation, i.e., pausing the data, send out
> commits (as of the current implementation), and send an event to the
> polling thread to invoke the callback
>   2. During: Waiting for the callback to be triggered.  Meanwhile, the
> background thread will continue to poll the channel, until the ack event
> arrives
>   3. After: Once the invocation ack event is received, we move on to
> request to join the group.
>
> Maybe you are right, perhaps, we don't need the PARTITION_REVOKED state.
>
> >
> > 3. How could we handle the Subscription state change in poll thread
> earlier
> > than REVOKE_PARTITION/ASSIGN_PARTITION events arrived?
> > Suppose, background thread sending REVOKE_PARTITION event to poll thread,
> > and before handing it, the consumer updates the subscription.
> > So in this situation, we'll still invoke revoke callback or not?
> > This won't happen in current design because they are handing in one
> thread.
> >
>
> Thanks for bringing this to the table. Would it make sense to lock the
> subscriptionState in some scenarios like this? In this case, this requires
> API changes, like throwing an exception to tell the users that the
> revocation is happening.
>
> >
> > 4. *Is there a better way to configure session interval and heartbeat
> > interval?*
> > These configs are moved to broker side in KIP-848 IIRC.
> > Maybe you can check that KIP and update here.
> >
> Thanks.
>
> >
> > Some typos and hard to understand
> >
> Hmm, terribly sorry about these typos.  Turned out I didn't publish the
> updated copy.  Will address them and update the document right away.
>
>
> > 5. Firstly, it _mcomplexfixes_ increasingly difficult ???
> > 6. it simplifies the _pd_ design ? What is pd?
> > 7. In "Background thread and its lifecycle" section, I guess the 2 points
> > should be 3c and 3d, right?
> > That is:
> > a. Check if there is an in-flight event. If not, poll for the new events
> > from 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1827

[Apache Jenkins Server]

See 

[DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi all,

I'd like to start a discussion for the KIP-926: introducing
acks=min.insync.replicas config. This KIP is to introduce
`acks=min.insync.replicas` config value in producer, to improve the write
throughput and still guarantee high durability.

Please check the link for more detail:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config

Any feedback is welcome.

Thank you.
Luke

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Randall Hauch]

Hi, Mickael.

Would it be okay for me to merge the commit from
https://github.com/apache/kafka/pull/13688 to the 3.5 branch? This brings
back two methods to maintain backward compatibility to the KafkaBasedLog
class (https://issues.apache.org/jira/browse/KAFKA-14974), which were
refactored in Feb as part of
https://issues.apache.org/jira/browse/KAFKA-14455. It isn't in Connect's
public API, but has been used outside of Connect and we have always tried
to preserve backward compatibility whenever possible. This fix has already
been applied to trunk (https://github.com/apache/kafka/pull/13688) and
cherry-picked to the 3.4 and 3.3 branches, and it'd be great to get this
into 3.5.0 before it is released with the binary-incompatible changes.

This change simply renames the two methods whose signatures were changed
recently and the few places where those methods are used, and then adds
back the methods with the previous signature. It does not change other
functionality, and IMO the risk is low.

Thanks,

Randall


On Sat, May 6, 2023 at 10:31 AM David Arthur  wrote:

> I resolved these three:
> * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix version
> * KAFKA-14805 is merged to trunk and 3.5
> * KAFKA-14918 is merged to trunk and 3.5
>
> KAFKA-14692 (docs issue) is still a not done
>
> Looks like KAFKA-14084 is now resolved as well (it's in trunk and 3.5).
>
> I'll try to find out about KAFKA-14698, I think it's likely a WONTFIX.
>
> -David
>
> On Fri, May 5, 2023 at 10:43 AM Mickael Maison 
> wrote:
>
> > Hi David,
> >
> > Thanks for the update!
> > You still own 4 other tickets targeting 3.5: KAFKA-14840, KAFKA-14805,
> > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > release?
> > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > migration. Should I move them too?
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 5, 2023 at 4:27 PM David Arthur
> >  wrote:
> > >
> > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > >
> > > Cheers,
> > > David
> > >
> > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Divij,
> > > >
> > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> > > > updated since 3.4.
> > > > Regarding your PR, I would have been in favor of bringing this to 3.5
> > > > a couple of weeks ago, but we're now a week past code freeze for 3.5.
> > > > Apart if this fixes CVEs, or significant bugs, I think we should only
> > > > merge it in trunk.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya  >
> > > > wrote:
> > > > >
> > > > > Hey Mickael
> > > > >
> > > > > Should we consider performing an update of the minor versions of
> the
> > > > > dependencies in 3.5 (per
> https://github.com/apache/kafka/pull/13673
> > )?
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen 
> > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > There are 1 bug and 1 improvement that I'd like to backport to
> > 3.5.
> > > > > > > 1. A small improvement for ZK migration based on KAFKA-14840
> > > > (mentioned
> > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > >
> > > > > > > 2. A bug will cause the KRaft controller node to shut down
> > > > unexpectedly.
> > > > > > PR
> > > > > > > is ready for review.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > > > https://github.com/apache/kafka/pull/13653
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > Yes you can backport these to 3.5. Let me know when you are
> > done.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Thu, Apr 27, 2023 at 9:02 PM David Arthur
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hey Mickael,
> > > > > > > > >
> > > > > > > > > I have one major ZK migration improvement (KAFKA-14805)
> that
> > > > landed
> > > > > > in
> > > > > > > > > trunk this week that I'd like to merge to 3.5 (once we fix
> > some
> > > > test
> > > > > > > > > failures it introduced). After that, I have another PR for
> > > > > > KAFKA-14840
> > > > > > > > > which is essentially a huge bug in the ZK migration logic
> > that
> > > > needs
> > > > > > to
> > > > > > > > > land in 3.5.
> > > > > > > > 

[jira] [Created] (KAFKA-14980) MirrorMaker consumers don't get configs prefixed with source.cluster

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14980:
--

 Summary: MirrorMaker consumers don't get configs prefixed with 
source.cluster
 Key: KAFKA-14980
 URL: https://issues.apache.org/jira/browse/KAFKA-14980
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.5.0
Reporter: Mickael Maison


As part of KAFKA-14021, we made a change to 
MirrorConnectorConfig.sourceConsumerConfig() to grab all configs that start 
with "source.". Previously it was grabbing configs prefixed with 
"source.cluster.". 

This means existing connector configuration stop working, as configurations 
such as bootstrap.servers are not passed to source consumers.

For example, the following connector configuration was valid in 3.4 and now 
makes the connector tasks fail:

{code:json}
{
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"name": "source",
"topics": "test",
"tasks.max": "30",
"source.cluster.alias": "one",
"target.cluster.alias": "two",
"source.cluster.bootstrap.servers": "localhost:9092",
   "target.cluster.bootstrap.servers": "localhost:29092"
}
{code}


The connector attempts to start source consumers with bootstrap.servers = [] 
and the task crash with 


{noformat}
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at 
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:837)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:671)
at 
org.apache.kafka.connect.mirror.MirrorUtils.newConsumer(MirrorUtils.java:59)
at 
org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:103)
at 
org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:274)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
at 
org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
at 
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable 
bootstrap urls given in bootstrap.servers
{noformat}






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14979) Incorrect lag was calculated when markPartitionsForTruncation in ReplicaAlterLogDirsThread

[hudeqi (Jira)]

hudeqi created KAFKA-14979:
--

 Summary: Incorrect lag was calculated when 
markPartitionsForTruncation in ReplicaAlterLogDirsThread
 Key: KAFKA-14979
 URL: https://issues.apache.org/jira/browse/KAFKA-14979
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.3.2
Reporter: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14978) ExactlyOnceWorkerSourceTask does not remove parent metrics

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-14978:


 Summary: ExactlyOnceWorkerSourceTask does not remove parent metrics
 Key: KAFKA-14978
 URL: https://issues.apache.org/jira/browse/KAFKA-14978
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Daniel Urban
Assignee: Daniel Urban


ExactlyOnceWorkerSourceTask removeMetrics does not invoke super.removeMetrics, 
meaning that only the transactional metrics are removed, and common source task 
metrics are not.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)