Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Colov/Divij for adding the KIP-952. I do not think it is a
blocker for 3.6.0. We can discuss the KIP in the respective thread.

~Satish.

On Sun, 23 Jul 2023 at 07:21, Satish Duggana  wrote:
>
> Thanks ShunKang for the update. I added both the KIPs to the wiki.
> Please feel free to update the wiki with the latest.
>
> ~Satish.
>
> On Sat, 22 Jul 2023 at 22:50, ShunKang Lin  wrote:
> >
> > Hi Satish,
> >
> > Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory copy" [1]
> > and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
> > copying" [2] to the release plan?
> > Thanks!
> >
> > [1]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
> >
> > [2]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> > I would appreciate a few more reviews on the pull request (
> > https://github.com/apache/kafka/pull/12685) for KIP-872.
> >
> > Best,
> > ShunKang
> >
> > Divij Vaidya  于2023年7月22日周六 20:06写道：
> >
> > > Hi Satish
> > >
> > > I have added the following accepted KIPs to the release plan. Please let 
> > > me
> > > know if something requires a change.
> > >
> > > Accepted KIPs -
> > >
> > > 1.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> > >
> > > 2.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> > >
> > >
> > > Pending discussion KIP which I believe is important to be merged into 3.6 
> > > -
> > >
> > > 3.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > >
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> > > wrote:
> > >
> > > > Thanks Hao for the update on KIP-925.
> > > >
> > > > On Thu, 20 Jul 2023 at 23:05, Hao Li  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > KIP-925 was accepted and currently under implementation. I just added
> > > it
> > > > to
> > > > > the release plan.
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > > > >
> > > > > Thanks,
> > > > > Hao
> > > > >
> > > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > > > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > A couple of days ago I opened a new KIP for discussion - KIP-952
> > > [1]. I
> > > > > > believe it might be a blocker for the release of 3.6.0, but I wanted
> > > to
> > > > > > bring it up here for a decision on its urgency with the current set
> > > of
> > > > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij)
> > > > given
> > > > > > that the date for KIP freeze is fast approaching.
> > > > > > What are your thoughts on the matter?
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > > > >
> > > > > > Best,
> > > > > > Christo
> > > > > >
> > > > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Yash,
> > > > > > > Thanks for the update. Added KIP-793 to the release plan. Please
> > > feel
> > > > > > > free to update the release wiki with any other updates on the KIP.
> > > > > > >
> > > > > > > ~Satish.
> > > > > > >
> > > > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > KIP-793 [1] just passed voting and we should be able to wrap up
> > > the
> > > > > > > > implementation in time for the 3.6.0 feature freeze. Could we 
> > > > > > > > add
> > > > it to
> > > > > > > the
> > > > > > > > release plan?
> > > > > > > >
> > > > > > > > [1] -
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yash
> > > > > > > >
> > > > > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > > > > satish.dugg...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I have created a release plan for Apache Kafka version 3.6.0 
> > > > > > > > > on
> > > > the
> > > > > > > > > wiki. You can access the release plan and all related
> > > > information by
> > > > > > > > > following this link:
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > > > > > > >
> > > > > > > > > The release plan outlines the key milestones and important
> > > dates
> > > > for
> > > > > > > > > version 3.6.0. Currently, the following dates have been set 
> > > > > > > > 

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks ShunKang for the update. I added both the KIPs to the wiki.
Please feel free to update the wiki with the latest.

~Satish.

On Sat, 22 Jul 2023 at 22:50, ShunKang Lin  wrote:
>
> Hi Satish,
>
> Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory copy" [1]
> and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
> copying" [2] to the release plan?
> Thanks!
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
>
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> I would appreciate a few more reviews on the pull request (
> https://github.com/apache/kafka/pull/12685) for KIP-872.
>
> Best,
> ShunKang
>
> Divij Vaidya  于2023年7月22日周六 20:06写道：
>
> > Hi Satish
> >
> > I have added the following accepted KIPs to the release plan. Please let me
> > know if something requires a change.
> >
> > Accepted KIPs -
> >
> > 1.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
> >
> > 2.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
> >
> >
> > Pending discussion KIP which I believe is important to be merged into 3.6 -
> >
> > 3.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Hao for the update on KIP-925.
> > >
> > > On Thu, 20 Jul 2023 at 23:05, Hao Li  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > KIP-925 was accepted and currently under implementation. I just added
> > it
> > > to
> > > > the release plan.
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > > >
> > > > Thanks,
> > > > Hao
> > > >
> > > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > A couple of days ago I opened a new KIP for discussion - KIP-952
> > [1]. I
> > > > > believe it might be a blocker for the release of 3.6.0, but I wanted
> > to
> > > > > bring it up here for a decision on its urgency with the current set
> > of
> > > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij)
> > > given
> > > > > that the date for KIP freeze is fast approaching.
> > > > > What are your thoughts on the matter?
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Yash,
> > > > > > Thanks for the update. Added KIP-793 to the release plan. Please
> > feel
> > > > > > free to update the release wiki with any other updates on the KIP.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> > > wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > KIP-793 [1] just passed voting and we should be able to wrap up
> > the
> > > > > > > implementation in time for the 3.6.0 feature freeze. Could we add
> > > it to
> > > > > > the
> > > > > > > release plan?
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yash
> > > > > > >
> > > > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > I have created a release plan for Apache Kafka version 3.6.0 on
> > > the
> > > > > > > > wiki. You can access the release plan and all related
> > > information by
> > > > > > > > following this link:
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > > > > > >
> > > > > > > > The release plan outlines the key milestones and important
> > dates
> > > for
> > > > > > > > version 3.6.0. Currently, the following dates have been set for
> > > the
> > > > > > > > release:
> > > > > > > >
> > > > > > > > KIP Freeze: 26th July 23
> > > > > > > > Feature Freeze : 16th Aug 23
> > > > > > > > Code Freeze : 30th Aug 23
> > > > > > > >
> > > > > > > > Please review the release plan and provide any additional
> > > information
> > > > > > > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > > > > > > authored any KIPs that are missing a status or if there are
> > > incorrect
> > > > > > > > status details, please make the necessary updates and inform me
> > > so
> > > 

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Divij for adding the KIPs to the wiki.

On Sat, 22 Jul 2023 at 17:36, Divij Vaidya  wrote:
>
> Hi Satish
>
> I have added the following accepted KIPs to the release plan. Please let me
> know if something requires a change.
>
> Accepted KIPs -
>
> 1.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
>
> 2.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
>
>
> Pending discussion KIP which I believe is important to be merged into 3.6 -
>
> 3.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
>
>
> --
> Divij Vaidya
>
>
>
> On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> wrote:
>
> > Thanks Hao for the update on KIP-925.
> >
> > On Thu, 20 Jul 2023 at 23:05, Hao Li  wrote:
> > >
> > > Hi Satish,
> > >
> > > KIP-925 was accepted and currently under implementation. I just added it
> > to
> > > the release plan.
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > >
> > > Thanks,
> > > Hao
> > >
> > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > A couple of days ago I opened a new KIP for discussion - KIP-952 [1]. I
> > > > believe it might be a blocker for the release of 3.6.0, but I wanted to
> > > > bring it up here for a decision on its urgency with the current set of
> > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij)
> > given
> > > > that the date for KIP freeze is fast approaching.
> > > > What are your thoughts on the matter?
> > > >
> > > > [1]
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > >
> > > > Best,
> > > > Christo
> > > >
> > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana 
> > > > wrote:
> > > >
> > > > > Hi Yash,
> > > > > Thanks for the update. Added KIP-793 to the release plan. Please feel
> > > > > free to update the release wiki with any other updates on the KIP.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> > wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > KIP-793 [1] just passed voting and we should be able to wrap up the
> > > > > > implementation in time for the 3.6.0 feature freeze. Could we add
> > it to
> > > > > the
> > > > > > release plan?
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > > >
> > > > > > Thanks,
> > > > > > Yash
> > > > > >
> > > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > > satish.dugg...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I have created a release plan for Apache Kafka version 3.6.0 on
> > the
> > > > > > > wiki. You can access the release plan and all related
> > information by
> > > > > > > following this link:
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > > > > >
> > > > > > > The release plan outlines the key milestones and important dates
> > for
> > > > > > > version 3.6.0. Currently, the following dates have been set for
> > the
> > > > > > > release:
> > > > > > >
> > > > > > > KIP Freeze: 26th July 23
> > > > > > > Feature Freeze : 16th Aug 23
> > > > > > > Code Freeze : 30th Aug 23
> > > > > > >
> > > > > > > Please review the release plan and provide any additional
> > information
> > > > > > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > > > > > authored any KIPs that are missing a status or if there are
> > incorrect
> > > > > > > status details, please make the necessary updates and inform me
> > so
> > > > > > > that I can keep the plan accurate and up to date.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Satish.
> > > > > > >
> > > > > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Thanks for volunteering!
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for volunteering Satish. +1.
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > > I would like to volunteer as release manager for the next
> > > > > release,
> > > > > > > > > > which will be Apache Kafka 3.6.0.
> > > > > > > > > >
> > > > > > > > > > If there are no objections, I will start a release plan a
> > 

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[David Jacot]

Hi Erik,

Thanks for the KIP. I would like to better understand the motivation of
this KIP. I am not familiar with async runtimes so please excuse me if I
ask stupid questions.

Could you elaborate a bit more on why the callbacks must be ran in another
thread vs in the invoker thread? This is not clear to me. In the example
that you use with the ConsumerRebalanceListener, I would have thought that
calling commitSync (without changing thread) would have achieved the same.
The invoker has to wait anyway on the offset commit completion so using
another thread does not bring any benefit here.  I suppose that I am
missing something here…

Regarding Chris’ proposal, this feels like a hack to me. The issue with it
is that we cannot guarantee it in the long term if it is not part of *the*
Consumer interface.

I second what Chris said. We are all trying to understand the motivation in
order to find a good solution for Kafka. I apologize if this creates
frustration. This is definitely not our goal.

Best,
David

PS: I just saw that you opened a new KIP based on Chris’ idea. This is not
necessary. You can just update the current KIP based on the discussion.

Le sam. 22 juil. 2023 à 18:34, Erik van Oosten 
a écrit :

> Colin, Matthias, Chris,
>
> I have expanded the use case description in KIP-944. I hope it is more
> clear what we're trying to achieve.
>
> https://cwiki.apache.org/confluence/x/chw0Dw
>
> Kind regards,
>  Erik.
>
>
> Op 22-07-2023 om 17:23 schreef Erik van Oosten:
> > Hello Chris,
> >
> > Thanks for elaborating Matthias' words. Apparently the use case
> > description is too terse. Indeed, that is not FUD and that is
> > something I can work with.
> >
> > > It's also worth mentioning that what's proposed in the KIP is only
> > blocked by the private access modifier on the KafkaConsumer::acquire
> > and KafkaConsumer::release methods. If we upgraded the visibility of
> > these methods from private to protected, it would be possible for
> > subclasses to implement the proposal in KIP-944, without any KIPs or
> > other changes to the official Java clients library.
> >
> > That is absolutely brilliant! Since I am pretty sure I am using the
> > consumer correctly, I could replace acquire and release with an empty
> > method body and be done.
> >
> > /Is making acquire and release protected something that other people
> > can live with?/
> > If yes, I will create a new PR with just that change.
> >
> > Kind regards,
> > Erik.
> >
> >
> > Op 22-07-2023 om 16:39 schreef Chris Egerton:
> >> Hi Erik,
> >>
> >> I don't think Matthias is bringing FUD to the discussion. Many of the
> >> people who maintain Kafka are familiar with Kafka client internals
> >> and the
> >> Java programming language, but not necessarily other JVM languages or
> >> asynchronous runtimes. I think it's reasonable to ask for a code
> >> snippet or
> >> two that demonstrates what you'd like to do with the consumer today that
> >> you can't because of restrictions around concurrent access, and this
> >> is not
> >> already addressed in the KIP. Linking to a docs page on Kotlin
> >> coroutines
> >> is helpful but still requires reviewers to gather a lot of context on
> >> their
> >> own that could more easily be provided in the KIP, and although the
> >> description of KAFKA-7143 is more detailed, I find it a little hard to
> >> follow as someone who isn't already familiar with the environment the
> >> user
> >> is working in.
> >>
> >> It's also worth mentioning that what's proposed in the KIP is only
> >> blocked
> >> by the private access modifier on the KafkaConsumer::acquire and
> >> KafkaConsumer::release methods. If we upgraded the visibility of these
> >> methods from private to protected, it would be possible for
> >> subclasses to
> >> implement the proposal in KIP-944, without any KIPs or other changes
> >> to the
> >> official Java clients library.
> >>
> >> Best,
> >>
> >> Chris
> >>
> >> On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
> >>   wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> I am getting a bit frustrated here. All the concerns and questions I
> >>> have seen so far are addressed in KIP-944.
> >>>
> >>> Please let me know if they are not clear enough, but please do not come
> >>> with FUD.
> >>>
> >>> Kind regards,
> >>>   Erik.
> >>>
> >>>
> >>> Op 21-07-2023 om 21:13 schreef Matthias J. Sax:
>  I am not a clients (or threading) expert, but I tend to agree to
>  Colin's concerns.
> 
>  In particular, it would be nice to see an example how you intent to
>  use the API (I am not familiar with Kotlin or it's co-routins), to
>  better understand what this changes help to solve to begin with.
> 
>  Opening up the consumer sounds potentially dangerous and we should
>  weight opportunity and risk before making a decision. So far, I see
>  risks but do not understand the opportunity you are after.
> 
> 
>  -Matthias
> 
>  On 7/14/23 11:43 AM, Kirk True 

[jira] [Created] (KAFKA-15236) Rename Remote Storage metrics to remove ambiguity

[Abhijeet Kumar (Jira)]

Abhijeet Kumar created KAFKA-15236:
--

 Summary: Rename Remote Storage metrics to remove ambiguity
 Key: KAFKA-15236
 URL: https://issues.apache.org/jira/browse/KAFKA-15236
 Project: Kafka
  Issue Type: Sub-task
Reporter: Abhijeet Kumar
Assignee: Abhijeet Kumar


As per the Tiered Storage feature introduced in 
[KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
 we added several metrics related to reads(from) and writes(to) for remote 
storage. The naming convention that was followed is confusing to the users.

For eg. in regular Kafka, BytesIn means bytes *_written_* to the log, and 
BytesOut means bytes *_read_* from the log. But with tiered storage, the 
concepts are reversed.
 * RemoteBytesIn means "Number of bytes *_read_* from remote storage per second"
 * RemoteBytesOut means "Number of bytes _*written*_ to remote storage per 
second"

We should rename the tiered storage related metrics to remove any ambiguity.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15235) No test coverage reports for Java due to settings for Jacoco being incompatible with Gradle 8.x

[Eike Thaden (Jira)]

Eike Thaden created KAFKA-15235:
---

 Summary: No test coverage reports for Java due to settings for 
Jacoco being incompatible with Gradle 8.x
 Key: KAFKA-15235
 URL: https://issues.apache.org/jira/browse/KAFKA-15235
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Affects Versions: 3.6.0
Reporter: Eike Thaden


On current dev branch, gradle 8.x fails while trying to generate test coverage 
reports as stated in the README, e.g. by running "./gradlew 
clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false". 
The error message states:

"Could not set unknown property 'enabled' for Report html of type 
org.gradle.api.reporting.internal.TaskGeneratedSingleDirectoryReport"

In "build.gradle", the library "jacoco" which is used to generate test coverage 
reports for the Java code is configured in two different places with these 
settings:

jacocoTestReport {
    dependsOn tasks.test
    sourceSets sourceSets.main
    reports {
        html.enabled = true
        xml.enabled = true
        csv.enabled = false
    }
}

With the latest version of jacoco, shipped with gradle 8.x, these config 
options are not compatible anymore. A correct configuration might look like 
like this:

jacocoTestReport {
    dependsOn tasks.test
    sourceSets sourceSets.main
    reports {
        html {
  required = true
    }
        xml {
  required = true
    }
        csv {
  required = false
    }
    }
}

However, even with these settings being accepted by Gradle, I was unable to 
generate any test coverage report. This might be due to some OOM issues, but I 
tried a lots of settings including increasing the maximum heap for the JVM 
gradle tasks without getting this to work.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-957 Support async runtimes in consumer

[Erik van Oosten]

Hello developers of the Java consumer,

This is a simpler alternative to KIP-944 as proposed by Chris Egerton.

In this proposal we make method acquire and release of the KafkaConsumer 
class protected. This allows anyone to implement these methods as 
appropriate for their environment.


The wiki page for KIP-957 contains more details 
https://cwiki.apache.org/confluence/x/lY6zDw


This is a call for discussion. If possible I would like to include this 
change in Kafka 3.6.

Any questions, comments, ideas and other additions are welcome!

Kind regards,
    Erik.


--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com

Re: Apache Kafka 3.6.0 release

[ShunKang Lin]

Hi Satish,

Could we add "KIP-863: Reduce CompletedFetch#parseRecord() memory copy" [1]
and "KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory
copying" [2] to the release plan?
Thanks!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035

[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
I would appreciate a few more reviews on the pull request (
https://github.com/apache/kafka/pull/12685) for KIP-872.

Best,
ShunKang

Divij Vaidya  于2023年7月22日周六 20:06写道：

> Hi Satish
>
> I have added the following accepted KIPs to the release plan. Please let me
> know if something requires a change.
>
> Accepted KIPs -
>
> 1.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier
>
> 2.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation
>
>
> Pending discussion KIP which I believe is important to be merged into 3.6 -
>
> 3.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
>
>
> --
> Divij Vaidya
>
>
>
> On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
> wrote:
>
> > Thanks Hao for the update on KIP-925.
> >
> > On Thu, 20 Jul 2023 at 23:05, Hao Li  wrote:
> > >
> > > Hi Satish,
> > >
> > > KIP-925 was accepted and currently under implementation. I just added
> it
> > to
> > > the release plan.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > >
> > > Thanks,
> > > Hao
> > >
> > > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > A couple of days ago I opened a new KIP for discussion - KIP-952
> [1]. I
> > > > believe it might be a blocker for the release of 3.6.0, but I wanted
> to
> > > > bring it up here for a decision on its urgency with the current set
> of
> > > > people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij)
> > given
> > > > that the date for KIP freeze is fast approaching.
> > > > What are your thoughts on the matter?
> > > >
> > > > [1]
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > > >
> > > > Best,
> > > > Christo
> > > >
> > > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana <
> satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Yash,
> > > > > Thanks for the update. Added KIP-793 to the release plan. Please
> feel
> > > > > free to update the release wiki with any other updates on the KIP.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> > wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > KIP-793 [1] just passed voting and we should be able to wrap up
> the
> > > > > > implementation in time for the 3.6.0 feature freeze. Could we add
> > it to
> > > > > the
> > > > > > release plan?
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > > >
> > > > > > Thanks,
> > > > > > Yash
> > > > > >
> > > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > > satish.dugg...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I have created a release plan for Apache Kafka version 3.6.0 on
> > the
> > > > > > > wiki. You can access the release plan and all related
> > information by
> > > > > > > following this link:
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > > > > >
> > > > > > > The release plan outlines the key milestones and important
> dates
> > for
> > > > > > > version 3.6.0. Currently, the following dates have been set for
> > the
> > > > > > > release:
> > > > > > >
> > > > > > > KIP Freeze: 26th July 23
> > > > > > > Feature Freeze : 16th Aug 23
> > > > > > > Code Freeze : 30th Aug 23
> > > > > > >
> > > > > > > Please review the release plan and provide any additional
> > information
> > > > > > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > > > > > authored any KIPs that are missing a status or if there are
> > incorrect
> > > > > > > status details, please make the necessary updates and inform me
> > so
> > > > > > > that I can keep the plan accurate and up to date.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Satish.
> > > > > > >
> > > > > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Thanks for volunteering!
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for 

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Colin, Matthias, Chris,

I have expanded the use case description in KIP-944. I hope it is more 
clear what we're trying to achieve.


https://cwiki.apache.org/confluence/x/chw0Dw

Kind regards,
    Erik.


Op 22-07-2023 om 17:23 schreef Erik van Oosten:

Hello Chris,

Thanks for elaborating Matthias' words. Apparently the use case 
description is too terse. Indeed, that is not FUD and that is 
something I can work with.


> It's also worth mentioning that what's proposed in the KIP is only 
blocked by the private access modifier on the KafkaConsumer::acquire 
and KafkaConsumer::release methods. If we upgraded the visibility of 
these methods from private to protected, it would be possible for 
subclasses to implement the proposal in KIP-944, without any KIPs or 
other changes to the official Java clients library.


That is absolutely brilliant! Since I am pretty sure I am using the 
consumer correctly, I could replace acquire and release with an empty 
method body and be done.


/Is making acquire and release protected something that other people 
can live with?/

If yes, I will create a new PR with just that change.

Kind regards,
    Erik.


Op 22-07-2023 om 16:39 schreef Chris Egerton:

Hi Erik,

I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals 
and the

Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code 
snippet or

two that demonstrates what you'd like to do with the consumer today that
you can't because of restrictions around concurrent access, and this 
is not
already addressed in the KIP. Linking to a docs page on Kotlin 
coroutines
is helpful but still requires reviewers to gather a lot of context on 
their

own that could more easily be provided in the KIP, and although the
description of KAFKA-7143 is more detailed, I find it a little hard to
follow as someone who isn't already familiar with the environment the 
user

is working in.

It's also worth mentioning that what's proposed in the KIP is only 
blocked

by the private access modifier on the KafkaConsumer::acquire and
KafkaConsumer::release methods. If we upgraded the visibility of these
methods from private to protected, it would be possible for 
subclasses to
implement the proposal in KIP-944, without any KIPs or other changes 
to the

official Java clients library.

Best,

Chris

On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
  wrote:


Hi Matthias,

I am getting a bit frustrated here. All the concerns and questions I
have seen so far are addressed in KIP-944.

Please let me know if they are not clear enough, but please do not come
with FUD.

Kind regards,
  Erik.


Op 21-07-2023 om 21:13 schreef Matthias J. Sax:

I am not a clients (or threading) expert, but I tend to agree to
Colin's concerns.

In particular, it would be nice to see an example how you intent to
use the API (I am not familiar with Kotlin or it's co-routins), to
better understand what this changes help to solve to begin with.

Opening up the consumer sounds potentially dangerous and we should
weight opportunity and risk before making a decision. So far, I see
risks but do not understand the opportunity you are after.


-Matthias

On 7/14/23 11:43 AM, Kirk True wrote:

Hi Erik,

Thanks for the KIP!

I empathize with your frustration over the radio silence. It gets
like that sometimes, and I apologize for my lack of feedback.

I’d personally like to see this lively exchange move over to the
DISCUSS thread you’d created before.

Thanks,
Kirk


On Jul 14, 2023, at 1:33 AM, Erik van Oosten
  wrote:

Hi Colin,

The way I understood Philp's message is that KIP-944 also plays nice
with KIP-945. But I might be mistaken.

Regardless, KIP-945 does /not/ resolve the underlying problem (the
need for nested consumer invocations) because it has the explicit
goal of not changing the user facing API.


... KIP-945 but haven't posted a DISCUSS thread yet

There is a thread called 'KafkaConsumer refactor proposal', but
indeed no official discussion yet.


I really don't want to be debugging complex interactions between
Java thread-local variables and green threads.

In that email thread, I proposed an API change in which callbacks
are no longer needed. The proposal completely removes the need for
such complex interactions. In addition, it gives clients the ability
to process at full speed even while a coorperative rebalance is
ongoing.

Regards,
  Erik.

Op 14-07-2023 om 00:36 schreef Colin McCabe:

HI Philip & Erik,

Hmm... if we agree that KIP-945 addresses this use case, I think it
would be better to just focus on that KIP. Fundamentally it's a
better and cleaner model than a complex scheme involving
thread-local variables. I really don't want to be debugging complex
interactions between Java thread-local variables and green threads.

It also generally helps to have some use-cases in mind when writing

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Chris Egerton]

Hi Erik,

I'm glad that this alternative is agreeable to you! Just to be clear
though, it does still technically require a KIP since it's still a change
to public API (just a much smaller one). And if we do decide to pursue this
approach, we'll have to define the expected contract for this method so
that it can be maintained correctly and documented for developers who may
otherwise be confused and misuse the method.

Cheers,

Chris

On Sat, Jul 22, 2023 at 12:02 PM Erik van Oosten
 wrote:

> Hi all,
>
> I have created https://github.com/apache/kafka/pull/14071 to implement
> Chris' idea.
>
> Kind regards,
>  Erik.
>
>
> Op 22-07-2023 om 16:39 schreef Chris Egerton:
> > Hi Erik,
> >
> > I don't think Matthias is bringing FUD to the discussion. Many of the
> > people who maintain Kafka are familiar with Kafka client internals and
> the
> > Java programming language, but not necessarily other JVM languages or
> > asynchronous runtimes. I think it's reasonable to ask for a code snippet
> or
> > two that demonstrates what you'd like to do with the consumer today that
> > you can't because of restrictions around concurrent access, and this is
> not
> > already addressed in the KIP. Linking to a docs page on Kotlin coroutines
> > is helpful but still requires reviewers to gather a lot of context on
> their
> > own that could more easily be provided in the KIP, and although the
> > description of KAFKA-7143 is more detailed, I find it a little hard to
> > follow as someone who isn't already familiar with the environment the
> user
> > is working in.
> >
> > It's also worth mentioning that what's proposed in the KIP is only
> blocked
> > by the private access modifier on the KafkaConsumer::acquire and
> > KafkaConsumer::release methods. If we upgraded the visibility of these
> > methods from private to protected, it would be possible for subclasses to
> > implement the proposal in KIP-944, without any KIPs or other changes to
> the
> > official Java clients library.
> >
> > Best,
> >
> > Chris
> >
> > On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
> >  wrote:
> >
> >> Hi Matthias,
> >>
> >> I am getting a bit frustrated here. All the concerns and questions I
> >> have seen so far are addressed in KIP-944.
> >>
> >> Please let me know if they are not clear enough, but please do not come
> >> with FUD.
> >>
> >> Kind regards,
> >>   Erik.
> >>
> >>
> >> Op 21-07-2023 om 21:13 schreef Matthias J. Sax:
> >>> I am not a clients (or threading) expert, but I tend to agree to
> >>> Colin's concerns.
> >>>
> >>> In particular, it would be nice to see an example how you intent to
> >>> use the API (I am not familiar with Kotlin or it's co-routins), to
> >>> better understand what this changes help to solve to begin with.
> >>>
> >>> Opening up the consumer sounds potentially dangerous and we should
> >>> weight opportunity and risk before making a decision. So far, I see
> >>> risks but do not understand the opportunity you are after.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 7/14/23 11:43 AM, Kirk True wrote:
>  Hi Erik,
> 
>  Thanks for the KIP!
> 
>  I empathize with your frustration over the radio silence. It gets
>  like that sometimes, and I apologize for my lack of feedback.
> 
>  I’d personally like to see this lively exchange move over to the
>  DISCUSS thread you’d created before.
> 
>  Thanks,
>  Kirk
> 
> > On Jul 14, 2023, at 1:33 AM, Erik van Oosten
> >  wrote:
> >
> > Hi Colin,
> >
> > The way I understood Philp's message is that KIP-944 also plays nice
> > with KIP-945. But I might be mistaken.
> >
> > Regardless, KIP-945 does /not/ resolve the underlying problem (the
> > need for nested consumer invocations) because it has the explicit
> > goal of not changing the user facing API.
> >
> >> ... KIP-945 but haven't posted a DISCUSS thread yet
> > There is a thread called 'KafkaConsumer refactor proposal', but
> > indeed no official discussion yet.
> >
> >> I really don't want to be debugging complex interactions between
> >> Java thread-local variables and green threads.
> > In that email thread, I proposed an API change in which callbacks
> > are no longer needed. The proposal completely removes the need for
> > such complex interactions. In addition, it gives clients the ability
> > to process at full speed even while a coorperative rebalance is
> > ongoing.
> >
> > Regards,
> >   Erik.
> >
> > Op 14-07-2023 om 00:36 schreef Colin McCabe:
> >> HI Philip & Erik,
> >>
> >> Hmm... if we agree that KIP-945 addresses this use case, I think it
> >> would be better to just focus on that KIP. Fundamentally it's a
> >> better and cleaner model than a complex scheme involving
> >> thread-local variables. I really don't want to be debugging complex
> >> interactions between Java thread-local variables and green 

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Hi all,

I have created https://github.com/apache/kafka/pull/14071 to implement 
Chris' idea.


Kind regards,
    Erik.


Op 22-07-2023 om 16:39 schreef Chris Egerton:

Hi Erik,

I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals and the
Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code snippet or
two that demonstrates what you'd like to do with the consumer today that
you can't because of restrictions around concurrent access, and this is not
already addressed in the KIP. Linking to a docs page on Kotlin coroutines
is helpful but still requires reviewers to gather a lot of context on their
own that could more easily be provided in the KIP, and although the
description of KAFKA-7143 is more detailed, I find it a little hard to
follow as someone who isn't already familiar with the environment the user
is working in.

It's also worth mentioning that what's proposed in the KIP is only blocked
by the private access modifier on the KafkaConsumer::acquire and
KafkaConsumer::release methods. If we upgraded the visibility of these
methods from private to protected, it would be possible for subclasses to
implement the proposal in KIP-944, without any KIPs or other changes to the
official Java clients library.

Best,

Chris

On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
 wrote:


Hi Matthias,

I am getting a bit frustrated here. All the concerns and questions I
have seen so far are addressed in KIP-944.

Please let me know if they are not clear enough, but please do not come
with FUD.

Kind regards,
  Erik.


Op 21-07-2023 om 21:13 schreef Matthias J. Sax:

I am not a clients (or threading) expert, but I tend to agree to
Colin's concerns.

In particular, it would be nice to see an example how you intent to
use the API (I am not familiar with Kotlin or it's co-routins), to
better understand what this changes help to solve to begin with.

Opening up the consumer sounds potentially dangerous and we should
weight opportunity and risk before making a decision. So far, I see
risks but do not understand the opportunity you are after.


-Matthias

On 7/14/23 11:43 AM, Kirk True wrote:

Hi Erik,

Thanks for the KIP!

I empathize with your frustration over the radio silence. It gets
like that sometimes, and I apologize for my lack of feedback.

I’d personally like to see this lively exchange move over to the
DISCUSS thread you’d created before.

Thanks,
Kirk


On Jul 14, 2023, at 1:33 AM, Erik van Oosten
 wrote:

Hi Colin,

The way I understood Philp's message is that KIP-944 also plays nice
with KIP-945. But I might be mistaken.

Regardless, KIP-945 does /not/ resolve the underlying problem (the
need for nested consumer invocations) because it has the explicit
goal of not changing the user facing API.


... KIP-945 but haven't posted a DISCUSS thread yet

There is a thread called 'KafkaConsumer refactor proposal', but
indeed no official discussion yet.


I really don't want to be debugging complex interactions between
Java thread-local variables and green threads.

In that email thread, I proposed an API change in which callbacks
are no longer needed. The proposal completely removes the need for
such complex interactions. In addition, it gives clients the ability
to process at full speed even while a coorperative rebalance is
ongoing.

Regards,
  Erik.

Op 14-07-2023 om 00:36 schreef Colin McCabe:

HI Philip & Erik,

Hmm... if we agree that KIP-945 addresses this use case, I think it
would be better to just focus on that KIP. Fundamentally it's a
better and cleaner model than a complex scheme involving
thread-local variables. I really don't want to be debugging complex
interactions between Java thread-local variables and green threads.

It also generally helps to have some use-cases in mind when writing
these things. If we get feedback about what would be useful for
async runtimes, that would probably help improve and focus KIP-945.
By the way, I can see you have a draft on the wiki for KIP-945 but
haven't posted a DISCUSS thread yet, so I assume it's not ready for
review yet ;)

best,
Colin


On Tue, Jul 11, 2023, at 12:24, Philip Nee wrote:

Hey Erik - Another thing I want to add to my comment is.  We are
in-process
of re-writing the KafkaConsumer, and I think your proposal would
work in
the new consumer because we are going to separate the user thread
and the
background thread.  Here is the 1-pager, and we are in process of
converting this in to KIP-945.

Thanks,
P

On Tue, Jul 11, 2023 at 10:33 AM Philip Nee 
wrote:


Hey Erik,

Sorry for holding up this email for a few days since Colin's
response
includes some of my concerns.  I'm in favor of this KIP, and I
think your
approach seems safe.  Of course, I probably missed something
therefore I
think this KIP needs to cover different use cases to demonstrate
it doesn't
cause any unsafe access. 

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Hello Chris,

Thanks for elaborating Matthias' words. Apparently the use case 
description is too terse. Indeed, that is not FUD and that is something 
I can work with.


> It's also worth mentioning that what's proposed in the KIP is only 
blocked by the private access modifier on the KafkaConsumer::acquire and 
KafkaConsumer::release methods. If we upgraded the visibility of these 
methods from private to protected, it would be possible for subclasses 
to implement the proposal in KIP-944, without any KIPs or other changes 
to the official Java clients library.


That is absolutely brilliant! Since I am pretty sure I am using the 
consumer correctly, I could replace acquire and release with an empty 
method body and be done.


/Is making acquire and release protected something that other people can 
live with?/

If yes, I will create a new PR with just that change.

Kind regards,
    Erik.


Op 22-07-2023 om 16:39 schreef Chris Egerton:

Hi Erik,

I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals and the
Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code snippet or
two that demonstrates what you'd like to do with the consumer today that
you can't because of restrictions around concurrent access, and this is not
already addressed in the KIP. Linking to a docs page on Kotlin coroutines
is helpful but still requires reviewers to gather a lot of context on their
own that could more easily be provided in the KIP, and although the
description of KAFKA-7143 is more detailed, I find it a little hard to
follow as someone who isn't already familiar with the environment the user
is working in.

It's also worth mentioning that what's proposed in the KIP is only blocked
by the private access modifier on the KafkaConsumer::acquire and
KafkaConsumer::release methods. If we upgraded the visibility of these
methods from private to protected, it would be possible for subclasses to
implement the proposal in KIP-944, without any KIPs or other changes to the
official Java clients library.

Best,

Chris

On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
  wrote:


Hi Matthias,

I am getting a bit frustrated here. All the concerns and questions I
have seen so far are addressed in KIP-944.

Please let me know if they are not clear enough, but please do not come
with FUD.

Kind regards,
  Erik.


Op 21-07-2023 om 21:13 schreef Matthias J. Sax:

I am not a clients (or threading) expert, but I tend to agree to
Colin's concerns.

In particular, it would be nice to see an example how you intent to
use the API (I am not familiar with Kotlin or it's co-routins), to
better understand what this changes help to solve to begin with.

Opening up the consumer sounds potentially dangerous and we should
weight opportunity and risk before making a decision. So far, I see
risks but do not understand the opportunity you are after.


-Matthias

On 7/14/23 11:43 AM, Kirk True wrote:

Hi Erik,

Thanks for the KIP!

I empathize with your frustration over the radio silence. It gets
like that sometimes, and I apologize for my lack of feedback.

I’d personally like to see this lively exchange move over to the
DISCUSS thread you’d created before.

Thanks,
Kirk


On Jul 14, 2023, at 1:33 AM, Erik van Oosten
  wrote:

Hi Colin,

The way I understood Philp's message is that KIP-944 also plays nice
with KIP-945. But I might be mistaken.

Regardless, KIP-945 does /not/ resolve the underlying problem (the
need for nested consumer invocations) because it has the explicit
goal of not changing the user facing API.


... KIP-945 but haven't posted a DISCUSS thread yet

There is a thread called 'KafkaConsumer refactor proposal', but
indeed no official discussion yet.


I really don't want to be debugging complex interactions between
Java thread-local variables and green threads.

In that email thread, I proposed an API change in which callbacks
are no longer needed. The proposal completely removes the need for
such complex interactions. In addition, it gives clients the ability
to process at full speed even while a coorperative rebalance is
ongoing.

Regards,
  Erik.

Op 14-07-2023 om 00:36 schreef Colin McCabe:

HI Philip & Erik,

Hmm... if we agree that KIP-945 addresses this use case, I think it
would be better to just focus on that KIP. Fundamentally it's a
better and cleaner model than a complex scheme involving
thread-local variables. I really don't want to be debugging complex
interactions between Java thread-local variables and green threads.

It also generally helps to have some use-cases in mind when writing
these things. If we get feedback about what would be useful for
async runtimes, that would probably help improve and focus KIP-945.
By the way, I can see you have a draft on the wiki for KIP-945 but
haven't posted a DISCUSS thread yet, so I assume it's not ready for
review 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2029

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 393204 lines...]

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreNullRecord() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromSourceTopic(boolean) > [2] false 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[1] true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldSuccessfullyStartWhenLoggingDisabled(boolean) > 
[2] false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [1] 
true PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
RestoreIntegrationTest > shouldRestoreStateFromChangelogTopic(boolean) > [2] 
false PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [1] true 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
SmokeTestDriverIntegrationTest > shouldWorkWithRebalance(boolean) > [2] false 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
SmokeTestDriverIntegrationTest > 

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Chris Egerton]

Hi Erik,

I don't think Matthias is bringing FUD to the discussion. Many of the
people who maintain Kafka are familiar with Kafka client internals and the
Java programming language, but not necessarily other JVM languages or
asynchronous runtimes. I think it's reasonable to ask for a code snippet or
two that demonstrates what you'd like to do with the consumer today that
you can't because of restrictions around concurrent access, and this is not
already addressed in the KIP. Linking to a docs page on Kotlin coroutines
is helpful but still requires reviewers to gather a lot of context on their
own that could more easily be provided in the KIP, and although the
description of KAFKA-7143 is more detailed, I find it a little hard to
follow as someone who isn't already familiar with the environment the user
is working in.

It's also worth mentioning that what's proposed in the KIP is only blocked
by the private access modifier on the KafkaConsumer::acquire and
KafkaConsumer::release methods. If we upgraded the visibility of these
methods from private to protected, it would be possible for subclasses to
implement the proposal in KIP-944, without any KIPs or other changes to the
official Java clients library.

Best,

Chris

On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten
 wrote:

> Hi Matthias,
>
> I am getting a bit frustrated here. All the concerns and questions I
> have seen so far are addressed in KIP-944.
>
> Please let me know if they are not clear enough, but please do not come
> with FUD.
>
> Kind regards,
>  Erik.
>
>
> Op 21-07-2023 om 21:13 schreef Matthias J. Sax:
> > I am not a clients (or threading) expert, but I tend to agree to
> > Colin's concerns.
> >
> > In particular, it would be nice to see an example how you intent to
> > use the API (I am not familiar with Kotlin or it's co-routins), to
> > better understand what this changes help to solve to begin with.
> >
> > Opening up the consumer sounds potentially dangerous and we should
> > weight opportunity and risk before making a decision. So far, I see
> > risks but do not understand the opportunity you are after.
> >
> >
> > -Matthias
> >
> > On 7/14/23 11:43 AM, Kirk True wrote:
> >> Hi Erik,
> >>
> >> Thanks for the KIP!
> >>
> >> I empathize with your frustration over the radio silence. It gets
> >> like that sometimes, and I apologize for my lack of feedback.
> >>
> >> I’d personally like to see this lively exchange move over to the
> >> DISCUSS thread you’d created before.
> >>
> >> Thanks,
> >> Kirk
> >>
> >>> On Jul 14, 2023, at 1:33 AM, Erik van Oosten
> >>>  wrote:
> >>>
> >>> Hi Colin,
> >>>
> >>> The way I understood Philp's message is that KIP-944 also plays nice
> >>> with KIP-945. But I might be mistaken.
> >>>
> >>> Regardless, KIP-945 does /not/ resolve the underlying problem (the
> >>> need for nested consumer invocations) because it has the explicit
> >>> goal of not changing the user facing API.
> >>>
>  ... KIP-945 but haven't posted a DISCUSS thread yet
> >>>
> >>> There is a thread called 'KafkaConsumer refactor proposal', but
> >>> indeed no official discussion yet.
> >>>
>  I really don't want to be debugging complex interactions between
>  Java thread-local variables and green threads.
> >>>
> >>> In that email thread, I proposed an API change in which callbacks
> >>> are no longer needed. The proposal completely removes the need for
> >>> such complex interactions. In addition, it gives clients the ability
> >>> to process at full speed even while a coorperative rebalance is
> >>> ongoing.
> >>>
> >>> Regards,
> >>>  Erik.
> >>>
> >>> Op 14-07-2023 om 00:36 schreef Colin McCabe:
>  HI Philip & Erik,
> 
>  Hmm... if we agree that KIP-945 addresses this use case, I think it
>  would be better to just focus on that KIP. Fundamentally it's a
>  better and cleaner model than a complex scheme involving
>  thread-local variables. I really don't want to be debugging complex
>  interactions between Java thread-local variables and green threads.
> 
>  It also generally helps to have some use-cases in mind when writing
>  these things. If we get feedback about what would be useful for
>  async runtimes, that would probably help improve and focus KIP-945.
>  By the way, I can see you have a draft on the wiki for KIP-945 but
>  haven't posted a DISCUSS thread yet, so I assume it's not ready for
>  review yet ;)
> 
>  best,
>  Colin
> 
> 
>  On Tue, Jul 11, 2023, at 12:24, Philip Nee wrote:
> > Hey Erik - Another thing I want to add to my comment is.  We are
> > in-process
> > of re-writing the KafkaConsumer, and I think your proposal would
> > work in
> > the new consumer because we are going to separate the user thread
> > and the
> > background thread.  Here is the 1-pager, and we are in process of
> > converting this in to KIP-945.
> >
> > Thanks,
> > P
> >
> > On Tue, Jul 11, 

Re: Apache Kafka 3.6.0 release

[Divij Vaidya]

Hi Satish

I have added the following accepted KIPs to the release plan. Please let me
know if something requires a change.

Accepted KIPs -

1.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier

2.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation


Pending discussion KIP which I believe is important to be merged into 3.6 -

3.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage


--
Divij Vaidya



On Sat, Jul 22, 2023 at 6:41 AM Satish Duggana 
wrote:

> Thanks Hao for the update on KIP-925.
>
> On Thu, 20 Jul 2023 at 23:05, Hao Li  wrote:
> >
> > Hi Satish,
> >
> > KIP-925 was accepted and currently under implementation. I just added it
> to
> > the release plan.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> >
> > Thanks,
> > Hao
> >
> > On Thu, Jul 20, 2023 at 6:18 AM Christo Lolov 
> > wrote:
> >
> > > Hello!
> > >
> > > A couple of days ago I opened a new KIP for discussion - KIP-952 [1]. I
> > > believe it might be a blocker for the release of 3.6.0, but I wanted to
> > > bring it up here for a decision on its urgency with the current set of
> > > people who are looking at Tiered Storage (Satish, Luke, Ivan, Divij)
> given
> > > that the date for KIP freeze is fast approaching.
> > > What are your thoughts on the matter?
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage
> > >
> > > Best,
> > > Christo
> > >
> > > On Sat, 8 Jul 2023 at 13:06, Satish Duggana 
> > > wrote:
> > >
> > > > Hi Yash,
> > > > Thanks for the update. Added KIP-793 to the release plan. Please feel
> > > > free to update the release wiki with any other updates on the KIP.
> > > >
> > > > ~Satish.
> > > >
> > > > On Fri, 7 Jul 2023 at 10:52, Yash Mayya 
> wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > KIP-793 [1] just passed voting and we should be able to wrap up the
> > > > > implementation in time for the 3.6.0 feature freeze. Could we add
> it to
> > > > the
> > > > > release plan?
> > > > >
> > > > > [1] -
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs
> > > > >
> > > > > Thanks,
> > > > > Yash
> > > > >
> > > > > On Mon, Jun 12, 2023 at 3:52 PM Satish Duggana <
> > > satish.dugg...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I have created a release plan for Apache Kafka version 3.6.0 on
> the
> > > > > > wiki. You can access the release plan and all related
> information by
> > > > > > following this link:
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > > > > >
> > > > > > The release plan outlines the key milestones and important dates
> for
> > > > > > version 3.6.0. Currently, the following dates have been set for
> the
> > > > > > release:
> > > > > >
> > > > > > KIP Freeze: 26th July 23
> > > > > > Feature Freeze : 16th Aug 23
> > > > > > Code Freeze : 30th Aug 23
> > > > > >
> > > > > > Please review the release plan and provide any additional
> information
> > > > > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > > > > authored any KIPs that are missing a status or if there are
> incorrect
> > > > > > status details, please make the necessary updates and inform me
> so
> > > > > > that I can keep the plan accurate and up to date.
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > > > On Mon, 17 Apr 2023 at 21:17, Luke Chen 
> wrote:
> > > > > > >
> > > > > > > Thanks for volunteering!
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Luke
> > > > > > >
> > > > > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks for volunteering Satish. +1.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> > > > > > satish.dugg...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I would like to volunteer as release manager for the next
> > > > release,
> > > > > > > > > which will be Apache Kafka 3.6.0.
> > > > > > > > >
> > > > > > > > > If there are no objections, I will start a release plan a
> week
> > > > after
> > > > > > > > > 3.5.0 release(around early May).
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Satish.
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
>

[jira] [Resolved] (KAFKA-15194) Rename local tiered storage segment with offset as prefix for easy navigation

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15194.
--
Resolution: Fixed

> Rename local tiered storage segment with offset as prefix for easy navigation
> -
>
> Key: KAFKA-15194
> URL: https://issues.apache.org/jira/browse/KAFKA-15194
> Project: Kafka
>  Issue Type: Task
>Reporter: Kamal Chandraprakash
>Assignee: Owen C.H. Leung
>Priority: Minor
>  Labels: newbie
> Fix For: 3.6.0
>
>
> In LocalTieredStorage which is an implementation of RemoteStorageManager, 
> segments are saved with random UUID. This makes navigating to a particular 
> segment harder. To navigate a given segment by offset, prepend the offset 
> information to the segment filename.
> https://github.com/apache/kafka/pull/13837#discussion_r1258896009



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] divijvaidya commented on pull request #533: MINOR: Website changes for 3.5.1 release

[via GitHub]

divijvaidya commented on PR #533:
URL: https://github.com/apache/kafka-site/pull/533#issuecomment-1646564445

   @mimaison please take a look when you get a chance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] divijvaidya opened a new pull request, #533: MINOR: Website changes for 3.5.1 release

[via GitHub]

divijvaidya opened a new pull request, #533:
URL: https://github.com/apache/kafka-site/pull/533

   ## Changes:
   1. Modify the download section to add links to new version
   2. Modify the download section to move prior versions to archive links.
   3. Modify the upgrade section to add information about new version.
   4. Add blog for new version.
   5. Remove in-progress mentioned in CVE list.
   
   ## Testing
   Tested locally. Verified all links. Screenshots attached.
   
   ![Screenshot 2023-07-22 at 13 06 
16](https://github.com/apache/kafka-site/assets/71267/f300e353-21df-4940-8770-6f880925c824)
   ![Screenshot 2023-07-22 at 12 57 
33](https://github.com/apache/kafka-site/assets/71267/ebc4e647-0c57-4775-b748-bb67216ecfa7)
   ![Screenshot 2023-07-22 at 10 38 
01](https://github.com/apache/kafka-site/assets/71267/020c79e3-00b4-4a6a-9e5d-0e4029a3e157)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

[Erik van Oosten]

Hi Matthias,

I am getting a bit frustrated here. All the concerns and questions I 
have seen so far are addressed in KIP-944.


Please let me know if they are not clear enough, but please do not come 
with FUD.


Kind regards,
    Erik.


Op 21-07-2023 om 21:13 schreef Matthias J. Sax:
I am not a clients (or threading) expert, but I tend to agree to 
Colin's concerns.


In particular, it would be nice to see an example how you intent to 
use the API (I am not familiar with Kotlin or it's co-routins), to 
better understand what this changes help to solve to begin with.


Opening up the consumer sounds potentially dangerous and we should 
weight opportunity and risk before making a decision. So far, I see 
risks but do not understand the opportunity you are after.



-Matthias

On 7/14/23 11:43 AM, Kirk True wrote:

Hi Erik,

Thanks for the KIP!

I empathize with your frustration over the radio silence. It gets 
like that sometimes, and I apologize for my lack of feedback.


I’d personally like to see this lively exchange move over to the 
DISCUSS thread you’d created before.


Thanks,
Kirk

On Jul 14, 2023, at 1:33 AM, Erik van Oosten 
 wrote:


Hi Colin,

The way I understood Philp's message is that KIP-944 also plays nice 
with KIP-945. But I might be mistaken.


Regardless, KIP-945 does /not/ resolve the underlying problem (the 
need for nested consumer invocations) because it has the explicit 
goal of not changing the user facing API.



... KIP-945 but haven't posted a DISCUSS thread yet


There is a thread called 'KafkaConsumer refactor proposal', but 
indeed no official discussion yet.


I really don't want to be debugging complex interactions between 
Java thread-local variables and green threads.


In that email thread, I proposed an API change in which callbacks 
are no longer needed. The proposal completely removes the need for 
such complex interactions. In addition, it gives clients the ability 
to process at full speed even while a coorperative rebalance is 
ongoing.


Regards,
 Erik.

Op 14-07-2023 om 00:36 schreef Colin McCabe:

HI Philip & Erik,

Hmm... if we agree that KIP-945 addresses this use case, I think it 
would be better to just focus on that KIP. Fundamentally it's a 
better and cleaner model than a complex scheme involving 
thread-local variables. I really don't want to be debugging complex 
interactions between Java thread-local variables and green threads.


It also generally helps to have some use-cases in mind when writing 
these things. If we get feedback about what would be useful for 
async runtimes, that would probably help improve and focus KIP-945. 
By the way, I can see you have a draft on the wiki for KIP-945 but 
haven't posted a DISCUSS thread yet, so I assume it's not ready for 
review yet ;)


best,
Colin


On Tue, Jul 11, 2023, at 12:24, Philip Nee wrote:
Hey Erik - Another thing I want to add to my comment is.  We are 
in-process
of re-writing the KafkaConsumer, and I think your proposal would 
work in
the new consumer because we are going to separate the user thread 
and the

background thread.  Here is the 1-pager, and we are in process of
converting this in to KIP-945.

Thanks,
P

On Tue, Jul 11, 2023 at 10:33 AM Philip Nee  
wrote:



Hey Erik,

Sorry for holding up this email for a few days since Colin's 
response
includes some of my concerns.  I'm in favor of this KIP, and I 
think your
approach seems safe.  Of course, I probably missed something 
therefore I
think this KIP needs to cover different use cases to demonstrate 
it doesn't
cause any unsafe access. I think this can be demonstrated via 
diagrams and

some code in the KIP.

Thanks,
P

On Sat, Jul 8, 2023 at 12:28 PM Erik van Oosten
 wrote:


Hello Colin,

  >> In KIP-944, the callback thread can only delegate to 
another thread
after reading from and writing to a threadlocal variable, 
providing the

barriers right there.

  > I don't see any documentation that accessing thread local 
variables
provides a total store or load barrier. Do you have such 
documentation?

It seems like if this were the case, we could eliminate volatile
variables from most of the code base.

Now I was imprecise. The thread-locals are only somewhat 
involved. In

the KIP proposal the callback thread reads an access key from a
thread-local variable. It then needs to pass that access key to 
another
thread, which then can set it on its own thread-local variable. 
The act

of passing a value from one thread to another implies that a memory
barrier needs to be passed. However, this is all not so relevant 
since
there is no need to pass the access key back when the other 
thread is

done.

But now I think about it a bit more, the locking mechanism runs 
in a
synchronized block. If I remember correctly this should be 
enough to

pass read and write barriers.

  >> In the current implementation the consumer is also invoked 
from

random threads. If it works now, it should continue to work.
  > I'm not sure what you're 

[jira] [Created] (KAFKA-15234) Automate adding version to system tests

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15234:


 Summary: Automate adding version to system tests
 Key: KAFKA-15234
 URL: https://issues.apache.org/jira/browse/KAFKA-15234
 Project: Kafka
  Issue Type: Sub-task
Reporter: Divij Vaidya


In the release process, there is a step to add the release version to the 
system tests using the following steps. (search for "In trunk update the 
following files with the current release number." at 
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process)


_In trunk update the following files with the current release number. This is 
needed for a feature as well as a bug-fix release ([commit 
example|https://github.com/apache/kafka/commit/354db26b954c1df7c8c83748c466768399209b8c])_
 * _KAFKA-REPO-DIR/gradle/dependencies.gradle_
 * _KAFKA-REPO-DIR/tests/docker/Dockerfile_
 * _KAFKA-REPO-DIR/tests/kafkatest/version.py_
 * _KAFKA-REPO-DIR/vagrant/base.sh_



This task should create a python script that will do these automatically. For 
reference, similar code changes are performed using regEx at 
https://github.com/apache/kafka/blob/cc4e699d4cb2880d05603e2e8310d28e1c9f201a/release.py#L563



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2028

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 393925 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerLeft[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterInner[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testOuterOuter[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithRightVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testLeftWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TableTableJoinIntegrationTest > [caching enabled = false] > 
testInnerWithLeftVersionedOnly[caching enabled = false] PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskAssignorIntegrationTest > shouldProperlyConfigureTheAssignor PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskMetadataIntegrationTest > shouldReportCorrectEndOffsetInformation PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
TaskMetadataIntegrationTest > shouldReportCorrectCommittedOffsetInformation 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
EmitOnChangeIntegrationTest > shouldEmitSameRecordAfterFailover() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 184 > 
KStreamAggregationDedupIntegrationTest > 

Re: Request contributor permissions to kafka

[Boudjelda Mohamed Said]

Thank you so much

Best regards and have a great weekend


On Sat 22 Jul 2023 at 08:56, Luke Chen  wrote:

> Hi,
>
> Your account is all set.
>
> Thanks.
> Luke
>
> On Sat, Jul 22, 2023 at 2:17 AM Boudjelda Mohamed Said 
> wrote:
>
> > Hi
> > I would like to have a contributor permission to kafka, this is my
> jira
> > id: *bmscomp*
> >
> >
> > Best regards
> >
>

Re: Request contributor permissions to kafka

[Luke Chen]

Hi,

Your account is all set.

Thanks.
Luke

On Sat, Jul 22, 2023 at 2:17 AM Boudjelda Mohamed Said 
wrote:

> Hi
> I would like to have a contributor permission to kafka, this is my jira
> id: *bmscomp*
>
>
> Best regards
>