Re: [VOTE] KIP-859: Add Metadata Log Processing Error Related Metrics

[Niket Goel]

Thanks for taking the time to go over the KIP Colin.

While I agree with both your points about error handling, I think this KIP 
focuses on just exposing these errors via the proposed metrics and does not 
alter the error handling behavior on either the brokers or the controllers. The 
metrics (as proposed) are somewhat independent of that and should just hook 
into whatever we are doing. If we change the error handling then the metrics 
should be hooked into the new code as well.

Maybe the point you are trying to make is that having the properties which you 
mention below will generally make it so that errors are not committed to the 
log and hence having the metrics for the is sufficient (as the error will 
likely be transient and recoverable?).

Apologies if I did not understand the intent of your comment.

- Niket

> On Aug 2, 2022, at 3:34 PM, Colin McCabe  wrote:
> 
> Hi Niket,
> 
> Thanks for the KIP -- much appreciated! The new metrics look very useful.
> 
> I agree with the proposed error handling for errors on standby controllers 
> and brokers. For active controllers, I think we should establish two points:
> 
> 1. the active controller replays metadata before submitting it to the Raft 
> quorum
> 2. metadata replay errors on the active cause the process to exit, prior to 
> attempting to commit the record
> 
> This will allow most of these metadata replay errors to be noticed and NOT 
> committed to the metadata log, which I think will make things much more 
> robust. Since the controller process can be restarted very quickly, it 
> shouldn't be an undue operational burden. (It's true that when in combined 
> mode, restarts will take longer, but this kind of tradeoff is integral to 
> combined mode -- you get reduced fault isolation in exchange for the lower 
> overhead of one fewer JVM process).
> 
> best,
> Colin
> 
> 
> On Mon, Aug 1, 2022, at 18:05, David Arthur wrote:
>> Thanks, Niket.
>> 
>> +1 binding from me
>> 
>> -David
>> 
>> On Mon, Aug 1, 2022 at 8:15 PM Niket Goel  wrote:
>>> 
>>> Hi all,
>>> 
>>> I would like to start a vote on KIP-859 which adds some new metrics to 
>>> KRaft to allow for better visibility into log processing errors.
>>> 
>>> KIP 
>>> —ttps://cwiki.apache.org/confluence/display/KAFKA/KIP-859%3A+Add+Metadata+Log+Processing+Error+Related+Metrics
>>>  
>>> 
>>> Discussion Thread — 
>>> https://www.google.com/url?q=https://lists.apache.org/thread/yl87h1s484yc09yjo1no46hwpbv0qkwt=gmail-imap=166008449400=AOvVaw2R3Sj0u0NlQOG9XHh-Wgzs
>>> 
>>> Thanks
>>> Niket
>>> 

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Jorge Esteban Quilcate Otoya]

Congratulations, Sophie!

On Tue, 2 Aug 2022, 23:26 Colin McCabe,  wrote:

> Congratulations!
>
> best,
> Colin
>
>
> On Tue, Aug 2, 2022, at 12:09, Matthew Benedict de Detrich wrote:
> > Congratulations Sophie!
> >
> > --
> > Matthew de Detrich
> > Aiven Deutschland GmbH
> > Immanuelkirchstraße 26, 10405 Berlin
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > m: +491603708037
> > w: aiven.io e: matthew.dedetr...@aiven.io
> > On 2. Aug 2022, 01:42 +0200, Guozhang Wang , wrote:
> >> Hi everyone,
> >>
> >> I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> >> committer since Oct. 2020 and has been contributing to the community
> >> consistently, especially around Kafka Streams and Kafka java consumer.
> She
> >> has also presented about Kafka Streams at Kafka Summit London this
> year. It
> >> is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> >>
> >> Congratulations, Sophie!
> >>
> >> -- Guozhang Wang, on behalf of Apache Kafka PMC
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1115

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 497135 lines...]
[2022-08-02T22:49:58.495Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount STARTED
[2022-08-02T22:50:20.949Z] 
[2022-08-02T22:50:20.949Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargePartitionCount PASSED
[2022-08-02T22:50:20.949Z] 
[2022-08-02T22:50:20.949Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys STARTED
[2022-08-02T22:50:26.753Z] 
[2022-08-02T22:50:26.753Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorManyStandbys PASSED
[2022-08-02T22:50:26.753Z] 
[2022-08-02T22:50:26.753Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys STARTED
[2022-08-02T22:50:49.035Z] 
[2022-08-02T22:50:49.035Z] StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyStandbys PASSED
[2022-08-02T22:50:49.035Z] 
[2022-08-02T22:50:49.035Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers STARTED
[2022-08-02T22:50:49.035Z] 
[2022-08-02T22:50:49.035Z] StreamsAssignmentScaleTest > 
testFallbackPriorTaskAssignorLargeNumConsumers PASSED
[2022-08-02T22:50:49.035Z] 
[2022-08-02T22:50:49.035Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers STARTED
[2022-08-02T22:50:50.809Z] 
[2022-08-02T22:50:50.809Z] StreamsAssignmentScaleTest > 
testStickyTaskAssignorLargeNumConsumers PASSED
[2022-08-02T22:50:50.809Z] 
[2022-08-02T22:50:50.809Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() STARTED
[2022-08-02T22:50:53.469Z] 
[2022-08-02T22:50:53.469Z] AdjustStreamThreadCountTest > 
testConcurrentlyAccessThreads() PASSED
[2022-08-02T22:50:53.469Z] 
[2022-08-02T22:50:53.469Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() STARTED
[2022-08-02T22:50:58.304Z] 
[2022-08-02T22:50:58.304Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadReplacement() PASSED
[2022-08-02T22:50:58.304Z] 
[2022-08-02T22:50:58.304Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() STARTED
[2022-08-02T22:51:02.964Z] 
[2022-08-02T22:51:02.964Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveThreadsMultipleTimes() PASSED
[2022-08-02T22:51:02.964Z] 
[2022-08-02T22:51:02.964Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() STARTED
[2022-08-02T22:51:03.913Z] 
[2022-08-02T22:51:03.913Z] AdjustStreamThreadCountTest > 
shouldnNotRemoveStreamThreadWithinTimeout() PASSED
[2022-08-02T22:51:03.913Z] 
[2022-08-02T22:51:03.913Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() STARTED
[2022-08-02T22:51:26.198Z] 
[2022-08-02T22:51:26.198Z] AdjustStreamThreadCountTest > 
shouldAddAndRemoveStreamThreadsWhileKeepingNamesCorrect() PASSED
[2022-08-02T22:51:26.198Z] 
[2022-08-02T22:51:26.198Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() STARTED
[2022-08-02T22:51:27.972Z] 
[2022-08-02T22:51:27.972Z] AdjustStreamThreadCountTest > 
shouldAddStreamThread() PASSED
[2022-08-02T22:51:27.972Z] 
[2022-08-02T22:51:27.972Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() STARTED
[2022-08-02T22:51:32.631Z] 
[2022-08-02T22:51:32.631Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThreadWithStaticMembership() PASSED
[2022-08-02T22:51:32.631Z] 
[2022-08-02T22:51:32.631Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-08-02T22:51:36.247Z] 
[2022-08-02T22:51:36.247Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-08-02T22:51:36.247Z] 
[2022-08-02T22:51:36.247Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-08-02T22:51:40.908Z] 
[2022-08-02T22:51:40.908Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() PASSED
[2022-08-02T22:51:43.568Z] 
[2022-08-02T22:51:43.568Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 STARTED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 PASSED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() STARTED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() PASSED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() STARTED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() PASSED
[2022-08-02T22:51:44.516Z] 
[2022-08-02T22:51:44.516Z] FineGrainedAutoResetIntegrationTest > 

Re: [VOTE] KIP-859: Add Metadata Log Processing Error Related Metrics

[Colin McCabe]

Hi Niket,

Thanks for the KIP -- much appreciated! The new metrics look very useful.

I agree with the proposed error handling for errors on standby controllers and 
brokers. For active controllers, I think we should establish two points:

1. the active controller replays metadata before submitting it to the Raft 
quorum
2. metadata replay errors on the active cause the process to exit, prior to 
attempting to commit the record

This will allow most of these metadata replay errors to be noticed and NOT 
committed to the metadata log, which I think will make things much more robust. 
Since the controller process can be restarted very quickly, it shouldn't be an 
undue operational burden. (It's true that when in combined mode, restarts will 
take longer, but this kind of tradeoff is integral to combined mode -- you get 
reduced fault isolation in exchange for the lower overhead of one fewer JVM 
process).

best,
Colin


On Mon, Aug 1, 2022, at 18:05, David Arthur wrote:
> Thanks, Niket.
>
> +1 binding from me
>
> -David
>
> On Mon, Aug 1, 2022 at 8:15 PM Niket Goel  wrote:
>>
>> Hi all,
>>
>> I would like to start a vote on KIP-859 which adds some new metrics to KRaft 
>> to allow for better visibility into log processing errors.
>>
>> KIP 
>> —ttps://cwiki.apache.org/confluence/display/KAFKA/KIP-859%3A+Add+Metadata+Log+Processing+Error+Related+Metrics
>>  
>> 
>> Discussion Thread — 
>> https://lists.apache.org/thread/yl87h1s484yc09yjo1no46hwpbv0qkwt
>>
>> Thanks
>> Niket
>>

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Colin McCabe]

Congratulations!

best,
Colin


On Tue, Aug 2, 2022, at 12:09, Matthew Benedict de Detrich wrote:
> Congratulations Sophie!
>
> --
> Matthew de Detrich
> Aiven Deutschland GmbH
> Immanuelkirchstraße 26, 10405 Berlin
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> m: +491603708037
> w: aiven.io e: matthew.dedetr...@aiven.io
> On 2. Aug 2022, 01:42 +0200, Guozhang Wang , wrote:
>> Hi everyone,
>>
>> I'd like to introduce our new Kafka PMC member, Sophie. She has been a
>> committer since Oct. 2020 and has been contributing to the community
>> consistently, especially around Kafka Streams and Kafka java consumer. She
>> has also presented about Kafka Streams at Kafka Summit London this year. It
>> is my pleasure to announce that Sophie agreed to join the Kafka PMC.
>>
>> Congratulations, Sophie!
>>
>> -- Guozhang Wang, on behalf of Apache Kafka PMC

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Justine Olshan]

I've updated the KIP to make the new minimum value 1 and remove the -1
configuration.
I've also added the low priority to the configuration and edited the
description as Ismael mentioned.

I'm thinking about bringing this KIP to a vote soon! Let me know if there
are any other comments or questions.

Thanks,
Justine

On Tue, Aug 2, 2022 at 9:02 AM Jason Gustafson 
wrote:

> I agree with Ismael that we should remove -1. I think we tend to view the
> coupling of these behaviors into a single configuration as a mistake, so
> it's a little odd to keep it (even if in a weakened form).
>
> -Jason
>
> On Sat, Jul 30, 2022 at 7:37 AM Ismael Juma  wrote:
>
> > I would remove -1 altogether. Two more comments:
> >
> > 1. The current description kind of leads people towards aligning the
> config
> > with delivery.timeout.ms. Is that what we want? We could say it should
> be
> > higher than delivery.timeout.ms but indicate that the default is usually
> > fine. The main reason to reduce it would be to save memory, I guess.
> > 2. Each config has a priority, we should specify it for this one. I'm
> > assuming it will be "low".
> >
> > Ismael
> >
> > On Fri, Jul 29, 2022 at 2:38 PM Justine Olshan
> > 
> > wrote:
> >
> > > Hi all,
> > >
> > > I've updated the KIP to include the new default of 1 day and
> information
> > > about -1 in the description of the config.
> > > I wonder though if including -1 makes sense now that it is not the
> > default
> > > value. Is there a benefit for manually setting -1 vs manually setting
> the
> > > value that transactional.id.expiration.ms has?
> > >
> > > Let me know your thoughts.
> > > Thanks,
> > > Justine
> > >
> > > On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma 
> wrote:
> > >
> > > > +1 for having 1 day as the default and for including this change in
> the
> > > > release notes.
> > > >
> > > > Ismael
> > > >
> > > > On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson
> > >  > > > >
> > > > wrote:
> > > >
> > > > > I don't think a major release is a requirement for a default change
> > for
> > > > > what it's worth. I do appreciate that there is a preference for not
> > > > rocking
> > > > > the boat though. For a little bit of background here, the problem
> we
> > > > > have encountered in production since the idempotent producer became
> > the
> > > > > default is OOM errors due to huge numbers of producerIds that had
> to
> > be
> > > > > retained in the cache for 7 days. It is hard to prevent use cases
> > from
> > > > > emerging where producers are used and discarded rapidly. We will be
> > > > using a
> > > > > lower value for sure, but it would also be nice to reduce the
> > > likelihood
> > > > > for the community to see this problem. The benefit of the caching
> > > > > diminishes quickly over time since it is primarily meant to handle
> > > > producer
> > > > > retry windows. I do not think there is much difference between 1
> days
> > > > and 7
> > > > > days from an application perspective, but it is a huge difference
> for
> > > the
> > > > > broker's memory usage.
> > > > >
> > > > > Best,
> > > > > Jason
> > > > >
> > > > > On Wed, Jul 27, 2022 at 2:57 AM Sagar 
> > > wrote:
> > > > >
> > > > > > Thanks Justine for the KIP. I think it might be better to
> document
> > > the
> > > > > > correlation between the new config and delivery.timeout.ms in
> the
> > > > Public
> > > > > > Interfaces Description.
> > > > > >
> > > > > > Also, I agree with Luke that for now setting a default to -1
> should
> > > be
> > > > > > good. We can look to switch to 1 day with major release.
> > > > > >
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > >
> > > > > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen 
> > wrote:
> > > > > >
> > > > > > > Hi Justine,
> > > > > > >
> > > > > > > Thanks for the KIP.
> > > > > > > I agree with you that we should try our best to keep backward
> > > > > > > compatibility, although our intention is to have lower producer
> > id
> > > > > > > expiration timeout.
> > > > > > > So, I think we should keep default to -1 IMO.
> > > > > > > Maybe we change the default to 1 day in next major release
> (4.0)?
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for taking a look Jason!
> > > > > > > >
> > > > > > > > I wondered if we wanted to have a smaller default but wasn't
> > sure
> > > > > about
> > > > > > > the
> > > > > > > > compatibility story -- especially since there is the chance
> for
> > > > > > producer
> > > > > > > > IDs to expire silently.
> > > > > > > > I do think that 1 day is fairly reasonable. If I don't hear
> any
> > > > > > > conflicting
> > > > > > > > opinions I can go ahead and update the default.
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > > > > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > 

Re: Recent commit causes Gradle to load significantly slowly?

[Ismael Juma]

Does this happen consistently?

Ismael

On Tue, Aug 2, 2022 at 4:00 AM Matthew Benedict de Detrich
 wrote:

> After syncing my fork with upstream Kafka at some point in time in the
> past I noticed that Gradle takes significantly longer to load, i.e.
> spending 5+ minutes in the configuration phase when the Gradle runner
> starts.
>
> Is anyone else experiencing this problem?
>
> --
> Matthew de Detrich
> Aiven Deutschland GmbH
> Immanuelkirchstraße 26, 10405 Berlin
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> m: +491603708037
> w: aiven.io e: matthew.dedetr...@aiven.io
>

Re: Hosting Kafka Videos on ASF YouTube channel

[Bill Bejeck]

Hi Joe,

Yes, that is correct.  Sorry, I should have mentioned that in the original
email.  That is the only video where Tim says that.
The Kafka Streams videos do not mention Confluent.

We're currently pursuing editing the video to remove the "from Confluent"
part.
Note that the site also uses the same video on the "quickstart" page, so
both places will be fixed when editing is completed.

Can we pursue hosting the Kafka Streams videos for now, then revisit the
"What is Apache Kafka?" when the editing is done?

Thanks,
Bill


On Tue, Aug 2, 2022 at 3:12 PM Joe Brockmeier  wrote:

> Hi Bill,
>
> I'm not sure changing hosting would quite solve the problem. The first
> video I see on this page:
>
> https://kafka.apache.org/intro
>
> Starts with "Hi, I'm Bill Berglund from *Confluent*" rather than "Hi,
> I'm Bill from Apache Kafka" -- so moving to the ASF Youtube channel
> wouldn't completely solve the problem.
>
> On Tue, Aug 2, 2022 at 3:05 PM Bill Bejeck  wrote:
> >
> > Hi,
> >
> > I am an Apache Kafka® committer and PMC member, and I'm working on our
> site to address some issues around our embedded videos and branding.
> >
> > The Kafka site has six embedded videos:  https://kafka.apache.org/intro,
> https://kafka.apache.org/quickstart, and four videos on
> https://kafka.apache.org/32/documentation/streams/.
> >
> > The videos are hosted on the Confluent YouTube channel, so the branding
> on the video is from Confluent.  Since it's coming from YouTube, there's no
> way to change it.
> >
> > Would it be possible to upload these videos to the Apache Foundation
> YouTube channel (https://www.youtube.com/c/TheApacheFoundation/featured)?
> Doing this would automatically change the branding to Apache.
> >
> > Thanks, and I look forward to working with you on this matter.
> >
> > Bill Bejeck
>
>
>
> --
> Joe Brockmeier
> Vice President Marketing & Publicity
> j...@apache.org
>

Re: Hosting Kafka Videos on ASF YouTube channel

[Joe Brockmeier]

Hi Bill,

I'm not sure changing hosting would quite solve the problem. The first
video I see on this page:

https://kafka.apache.org/intro

Starts with "Hi, I'm Bill Berglund from *Confluent*" rather than "Hi,
I'm Bill from Apache Kafka" -- so moving to the ASF Youtube channel
wouldn't completely solve the problem.

On Tue, Aug 2, 2022 at 3:05 PM Bill Bejeck  wrote:
>
> Hi,
>
> I am an Apache Kafka® committer and PMC member, and I'm working on our site 
> to address some issues around our embedded videos and branding.
>
> The Kafka site has six embedded videos:  https://kafka.apache.org/intro, 
> https://kafka.apache.org/quickstart, and four videos on 
> https://kafka.apache.org/32/documentation/streams/.
>
> The videos are hosted on the Confluent YouTube channel, so the branding on 
> the video is from Confluent.  Since it's coming from YouTube, there's no way 
> to change it.
>
> Would it be possible to upload these videos to the Apache Foundation YouTube 
> channel (https://www.youtube.com/c/TheApacheFoundation/featured)?  Doing this 
> would automatically change the branding to Apache.
>
> Thanks, and I look forward to working with you on this matter.
>
> Bill Bejeck



-- 
Joe Brockmeier
Vice President Marketing & Publicity
j...@apache.org

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Matthew Benedict de Detrich]

Congratulations Sophie!

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io
On 2. Aug 2022, 01:42 +0200, Guozhang Wang , wrote:
> Hi everyone,
>
> I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> committer since Oct. 2020 and has been contributing to the community
> consistently, especially around Kafka Streams and Kafka java consumer. She
> has also presented about Kafka Streams at Kafka Summit London this year. It
> is my pleasure to announce that Sophie agreed to join the Kafka PMC.
>
> Congratulations, Sophie!
>
> -- Guozhang Wang, on behalf of Apache Kafka PMC

Re: [ANNOUNCE] Apache Kafka 3.2.1

[Matthew Benedict de Detrich]

Thanks for organising the release!

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io
On 2. Aug 2022, 01:46 +0200, David Arthur , wrote:
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.2.1
>
> This is a bugfix release with several fixes since the release of
> 3.2.0. A few of the major issues include:
>
> * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck
>
>
> All of the changes in this release can be found in the release notes:
>
> https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
>
> https://kafka.apache.org/downloads#3.2.1
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream of
> records to one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
> A big thank you for the following 19 contributors to this release!
>
> Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David
> Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma,
> James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen,
> Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor
> Somogyi-Vass, Walker Carlson
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
>
> Thank you!
>
> Regards,
> David Arthur

Hosting Kafka Videos on ASF YouTube channel

[Bill Bejeck]

Hi,

I am an Apache Kafka® committer and PMC member, and I'm working on our site
to address some issues around our embedded videos and branding.

The Kafka site has six embedded videos:  https://kafka.apache.org/intro,
https://kafka.apache.org/quickstart, and four videos on
https://kafka.apache.org/32/documentation/streams/.

The videos are hosted on the Confluent YouTube channel, so the branding on
the video is from Confluent.  Since it's coming from YouTube, there's no
way to change it.

Would it be possible to upload these videos to the Apache Foundation
YouTube channel (https://www.youtube.com/c/TheApacheFoundation/featured)
?  Doing this would
automatically change the branding to Apache.

Thanks, and I look forward to working with you on this matter.

Bill Bejeck

[DISCUSS] KIP-862: Implement self-join optimization

[Vasiliki Papavasileiou]

Hello everyone,

I would like to start the discussion for KIP-862: Implement self-join
optimization

The KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-862%3A+Self-join

Any suggestions are more than welcome.

Many thanks,
Vicky

[jira] [Resolved] (KAFKA-14122) Flaky test DynamicBrokerReconfigurationTest.testKeyStoreAlter

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14122.

Resolution: Fixed

> Flaky test DynamicBrokerReconfigurationTest.testKeyStoreAlter
> -
>
> Key: KAFKA-14122
> URL: https://issues.apache.org/jira/browse/KAFKA-14122
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core, system tests
>Reporter: Divij Vaidya
>Assignee: Divij Vaidya
>Priority: Major
>  Labels: flaky, flaky-test
> Fix For: 3.4.0
>
>
> CI Build: 
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12439/2/testReport/?cloudbees-analytics-link=scm-reporting%2Ftests%2Ffailed]
>  
> Failure log:
> {code:java}
> org.opentest4j.AssertionFailedError: Duplicates not expected ==> expected: 
>  but was: 
>   at 
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
>   at 
> app//org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:40)
>   at 
> app//org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:235)
>   at 
> app//kafka.server.DynamicBrokerReconfigurationTest.stopAndVerifyProduceConsume(DynamicBrokerReconfigurationTest.scala:1579)
>   at 
> app//kafka.server.DynamicBrokerReconfigurationTest.testKeyStoreAlter(DynamicBrokerReconfigurationTest.scala:399)
>   at 
> java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)
>   at 
> java.base@17.0.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>   at 
> java.base@17.0.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base@17.0.1/java.lang.reflect.Method.invoke(Method.java:568)
>   at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
>   at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
>   at 
> app//org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> app//org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-854 Separate configuration for producer ID expiry

[Jason Gustafson]

I agree with Ismael that we should remove -1. I think we tend to view the
coupling of these behaviors into a single configuration as a mistake, so
it's a little odd to keep it (even if in a weakened form).

-Jason

On Sat, Jul 30, 2022 at 7:37 AM Ismael Juma  wrote:

> I would remove -1 altogether. Two more comments:
>
> 1. The current description kind of leads people towards aligning the config
> with delivery.timeout.ms. Is that what we want? We could say it should be
> higher than delivery.timeout.ms but indicate that the default is usually
> fine. The main reason to reduce it would be to save memory, I guess.
> 2. Each config has a priority, we should specify it for this one. I'm
> assuming it will be "low".
>
> Ismael
>
> On Fri, Jul 29, 2022 at 2:38 PM Justine Olshan
> 
> wrote:
>
> > Hi all,
> >
> > I've updated the KIP to include the new default of 1 day and information
> > about -1 in the description of the config.
> > I wonder though if including -1 makes sense now that it is not the
> default
> > value. Is there a benefit for manually setting -1 vs manually setting the
> > value that transactional.id.expiration.ms has?
> >
> > Let me know your thoughts.
> > Thanks,
> > Justine
> >
> > On Fri, Jul 29, 2022 at 10:54 AM Ismael Juma  wrote:
> >
> > > +1 for having 1 day as the default and for including this change in the
> > > release notes.
> > >
> > > Ismael
> > >
> > > On Wed, Jul 27, 2022 at 9:16 AM Jason Gustafson
> >  > > >
> > > wrote:
> > >
> > > > I don't think a major release is a requirement for a default change
> for
> > > > what it's worth. I do appreciate that there is a preference for not
> > > rocking
> > > > the boat though. For a little bit of background here, the problem we
> > > > have encountered in production since the idempotent producer became
> the
> > > > default is OOM errors due to huge numbers of producerIds that had to
> be
> > > > retained in the cache for 7 days. It is hard to prevent use cases
> from
> > > > emerging where producers are used and discarded rapidly. We will be
> > > using a
> > > > lower value for sure, but it would also be nice to reduce the
> > likelihood
> > > > for the community to see this problem. The benefit of the caching
> > > > diminishes quickly over time since it is primarily meant to handle
> > > producer
> > > > retry windows. I do not think there is much difference between 1 days
> > > and 7
> > > > days from an application perspective, but it is a huge difference for
> > the
> > > > broker's memory usage.
> > > >
> > > > Best,
> > > > Jason
> > > >
> > > > On Wed, Jul 27, 2022 at 2:57 AM Sagar 
> > wrote:
> > > >
> > > > > Thanks Justine for the KIP. I think it might be better to document
> > the
> > > > > correlation between the new config and delivery.timeout.ms in the
> > > Public
> > > > > Interfaces Description.
> > > > >
> > > > > Also, I agree with Luke that for now setting a default to -1 should
> > be
> > > > > good. We can look to switch to 1 day with major release.
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > > On Wed, Jul 27, 2022 at 9:05 AM Luke Chen 
> wrote:
> > > > >
> > > > > > Hi Justine,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > > I agree with you that we should try our best to keep backward
> > > > > > compatibility, although our intention is to have lower producer
> id
> > > > > > expiration timeout.
> > > > > > So, I think we should keep default to -1 IMO.
> > > > > > Maybe we change the default to 1 day in next major release (4.0)?
> > > > > >
> > > > > > Thank you.
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Jul 27, 2022 at 7:13 AM Justine Olshan
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for taking a look Jason!
> > > > > > >
> > > > > > > I wondered if we wanted to have a smaller default but wasn't
> sure
> > > > about
> > > > > > the
> > > > > > > compatibility story -- especially since there is the chance for
> > > > > producer
> > > > > > > IDs to expire silently.
> > > > > > > I do think that 1 day is fairly reasonable. If I don't hear any
> > > > > > conflicting
> > > > > > > opinions I can go ahead and update the default.
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Jul 26, 2022 at 12:23 PM Jason Gustafson
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > >
> > > > > > > > Thanks for the KIP. Although I hate seeing new
> configurations,
> > I
> > > > > think
> > > > > > > this
> > > > > > > > is a good change. Combining these timeout behaviors into a
> > single
> > > > > > > > configuration was definitely a mistake, but we didn't
> > anticipate
> > > > the
> > > > > > > > problem with the producer id cache. I do wonder if we can
> make
> > > the
> > > > > > > default
> > > > > > > > a bit lower to reduce the chances that users will hit the
> same
> > > > memory
> > > > > > > > issues we have seen. After decoupling this configuration from
> > > > > > > > 

Re: [ANNOUNCE] Apache Kafka 3.2.1

[Bill Bejeck]

Thanks for running the release David!

-Bill

On Tue, Aug 2, 2022 at 4:34 AM Mickael Maison 
wrote:

> Thanks David for driving this release!
>
> On Tue, Aug 2, 2022 at 1:46 AM David Arthur 
> wrote:
> >
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 3.2.1
> >
> > This is a bugfix release with several fixes since the release of
> > 3.2.0. A few of the major issues include:
> >
> > * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> > * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> > * KAFKA-14024 Cooperative rebalance regression causing clients to get
> stuck
> >
> >
> > All of the changes in this release can be found in the release notes:
> >
> > https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release (Scala 2.12 and 2.13)
> from:
> >
> > https://kafka.apache.org/downloads#3.2.1
> >
> >
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> > ** The Producer API allows an application to publish a stream of
> > records to one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming
> > the input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide,
> > including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > Zalando, among others.
> >
> > A big thank you for the following 19 contributors to this release!
> >
> > Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David
> > Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma,
> > James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen,
> > Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor
> > Somogyi-Vass, Walker Carlson
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > https://kafka.apache.org/
> >
> >
> > Thank you!
> >
> > Regards,
> > David Arthur
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1114

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 563821 lines...]
[2022-08-02T15:12:05.074Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() STARTED
[2022-08-02T15:12:11.147Z] 
[2022-08-02T15:12:11.147Z] AdjustStreamThreadCountTest > 
shouldRemoveStreamThread() PASSED
[2022-08-02T15:12:11.147Z] 
[2022-08-02T15:12:11.147Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() STARTED
[2022-08-02T15:12:12.301Z] 
[2022-08-02T15:12:12.301Z] AdjustStreamThreadCountTest > 
shouldResizeCacheAfterThreadRemovalTimesOut() PASSED
[2022-08-02T15:12:13.672Z] 
[2022-08-02T15:12:13.672Z] RackAwarenessIntegrationTest > 
shouldDoRebalancingWithMaximumNumberOfClientTags() PASSED
[2022-08-02T15:12:13.672Z] 
[2022-08-02T15:12:13.672Z] RackAwarenessIntegrationTest > 
shouldDistributeStandbyReplicasWhenAllClientsAreLocatedOnASameClusterTag() 
STARTED
[2022-08-02T15:12:17.063Z] 
[2022-08-02T15:12:17.063Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 STARTED
[2022-08-02T15:12:17.249Z] 
[2022-08-02T15:12:17.249Z] RackAwarenessIntegrationTest > 
shouldDistributeStandbyReplicasWhenAllClientsAreLocatedOnASameClusterTag() 
PASSED
[2022-08-02T15:12:17.249Z] streams-3: SMOKE-TEST-CLIENT-CLOSED
[2022-08-02T15:12:17.249Z] streams-4: SMOKE-TEST-CLIENT-CLOSED
[2022-08-02T15:12:17.249Z] streams-0: SMOKE-TEST-CLIENT-CLOSED
[2022-08-02T15:12:17.249Z] streams-2: SMOKE-TEST-CLIENT-CLOSED
[2022-08-02T15:12:17.249Z] streams-1: SMOKE-TEST-CLIENT-CLOSED
[2022-08-02T15:12:18.168Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithGlobalAutoOffsetResetLatest()
 PASSED
[2022-08-02T15:12:18.169Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() STARTED
[2022-08-02T15:12:18.169Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingPattern() PASSED
[2022-08-02T15:12:18.169Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() STARTED
[2022-08-02T15:12:18.169Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldThrowExceptionOverlappingTopic() PASSED
[2022-08-02T15:12:18.169Z] 
[2022-08-02T15:12:18.169Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithInvalidCommittedOffsets() STARTED
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] FAILURE: Build failed with an exception.
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] * What went wrong:
[2022-08-02T15:12:21.858Z] Execution failed for task ':streams:unitTest'.
[2022-08-02T15:12:21.858Z] > Process 'Gradle Test Executor 122' finished with 
non-zero exit value 134
[2022-08-02T15:12:21.858Z]   This problem might be caused by incorrect test 
process configuration.
[2022-08-02T15:12:21.858Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.5/userguide/java_testing.html#sec:test_execution
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] * Try:
[2022-08-02T15:12:21.858Z] > Run with --stacktrace option to get the stack 
trace.
[2022-08-02T15:12:21.858Z] > Run with --info or --debug option to get more log 
output.
[2022-08-02T15:12:21.858Z] > Run with --scan to get full insights.
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] * Get more help at https://help.gradle.org
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] BUILD FAILED in 2h 24m 58s
[2022-08-02T15:12:21.858Z] 212 actionable tasks: 115 executed, 97 up-to-date
[2022-08-02T15:12:21.858Z] 
[2022-08-02T15:12:21.858Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2022-08-02-12-47-27.html
[2022-08-02T15:12:21.858Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.12
[2022-08-02T15:13:05.467Z] 
[2022-08-02T15:13:05.467Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithInvalidCommittedOffsets() PASSED
[2022-08-02T15:13:05.467Z] 
[2022-08-02T15:13:05.467Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithDefaultGlobalAutoOffsetResetEarliest()
 STARTED
[2022-08-02T15:13:54.676Z] 
[2022-08-02T15:13:54.676Z] FineGrainedAutoResetIntegrationTest > 
shouldOnlyReadRecordsWhereEarliestSpecifiedWithNoCommittedOffsetsWithDefaultGlobalAutoOffsetResetEarliest()
 PASSED
[2022-08-02T15:13:54.676Z] 

[GitHub] [kafka-site] bbejeck merged pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck merged PR #430:
URL: https://github.com/apache/kafka-site/pull/430


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

KAFKA-13648: KRaft ClusterInstance does not allow for deferred start

[Sayed Mohammad Hossein Torabi]

Dear Commiters,

Could you please Review this PR? https://github.com/apache/kafka/pull/12400
This PR, fixes the bug regarding Cluster Annotation in the tests and helps
the migration to Cluster annotation.

Kind regards,
Hossein Torabi

[GitHub] [kafka-site] divijvaidya commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

divijvaidya commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1202677727

   > Updated to include a warning about loading a video from a third party
   
   @bbejeck you might have forgotten to push the commit? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #430: MINOR: Add clickable images to load iframe videos

[GitBox]

bbejeck commented on PR #430:
URL: https://github.com/apache/kafka-site/pull/430#issuecomment-1202658336

   Updated to include a warning about loading a video from a third party
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Chris Egerton]

Congrats, Sophie!

On Mon, Aug 1, 2022 at 9:21 PM Luke Chen  wrote:

> Congrats Sophie! :)
>
> Luke
>
> On Tue, Aug 2, 2022 at 7:56 AM Adam Bellemare 
> wrote:
>
> > Congratulations Sophie! I’m glad to see you made as a PMC member! Well
> > earned.
> >
> > > On Aug 1, 2022, at 6:42 PM, Guozhang Wang  wrote:
> > >
> > > Hi everyone,
> > >
> > > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > > committer since Oct. 2020 and has been contributing to the community
> > > consistently, especially around Kafka Streams and Kafka java consumer.
> > She
> > > has also presented about Kafka Streams at Kafka Summit London this
> year.
> > It
> > > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> > >
> > > Congratulations, Sophie!
> > >
> > > -- Guozhang Wang, on behalf of Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Randall Hauch]

Congratulations, Sophie!

On Tue, Aug 2, 2022 at 8:09 AM Bill Bejeck  wrote:

> Congrats Sophie!
>
> -Bill
>
> On Tue, Aug 2, 2022 at 7:12 AM Tom Bentley  wrote:
>
> > Congratulations Sophie!
> >
> > On Tue, 2 Aug 2022 at 11:54, Robin Moffatt 
> > wrote:
> >
> > > Congrats Sophie, great news!
> > >
> > >
> > > --
> > >
> > > Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
> > @rmoff
> > >
> > >
> > > On Tue, 2 Aug 2022 at 00:42, Guozhang Wang  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to introduce our new Kafka PMC member, Sophie. She has been
> a
> > > > committer since Oct. 2020 and has been contributing to the community
> > > > consistently, especially around Kafka Streams and Kafka java
> consumer.
> > > She
> > > > has also presented about Kafka Streams at Kafka Summit London this
> > year.
> > > It
> > > > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> > > >
> > > > Congratulations, Sophie!
> > > >
> > > > -- Guozhang Wang, on behalf of Apache Kafka PMC
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Bill Bejeck]

Congrats Sophie!

-Bill

On Tue, Aug 2, 2022 at 7:12 AM Tom Bentley  wrote:

> Congratulations Sophie!
>
> On Tue, 2 Aug 2022 at 11:54, Robin Moffatt 
> wrote:
>
> > Congrats Sophie, great news!
> >
> >
> > --
> >
> > Robin Moffatt | Principal Developer Advocate | ro...@confluent.io |
> @rmoff
> >
> >
> > On Tue, 2 Aug 2022 at 00:42, Guozhang Wang  wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > > committer since Oct. 2020 and has been contributing to the community
> > > consistently, especially around Kafka Streams and Kafka java consumer.
> > She
> > > has also presented about Kafka Streams at Kafka Summit London this
> year.
> > It
> > > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> > >
> > > Congratulations, Sophie!
> > >
> > > -- Guozhang Wang, on behalf of Apache Kafka PMC
> > >
> >
>

[jira] [Created] (KAFKA-14134) Replace EasyMock with Mockito for WorkerConnectorTest

[Yash Mayya (Jira)]

Yash Mayya created KAFKA-14134:
--

 Summary: Replace EasyMock with Mockito for WorkerConnectorTest
 Key: KAFKA-14134
 URL: https://issues.apache.org/jira/browse/KAFKA-14134
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Yash Mayya






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14133) Remaining EasyMock to Mockito tests

[Christo Lolov (Jira)]

Christo Lolov created KAFKA-14133:
-

 Summary: Remaining EasyMock to Mockito tests
 Key: KAFKA-14133
 URL: https://issues.apache.org/jira/browse/KAFKA-14133
 Project: Kafka
  Issue Type: Sub-task
Reporter: Christo Lolov






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14132) Remaining PowerMock to Mockito tests

[Christo Lolov (Jira)]

Christo Lolov created KAFKA-14132:
-

 Summary: Remaining PowerMock to Mockito tests
 Key: KAFKA-14132
 URL: https://issues.apache.org/jira/browse/KAFKA-14132
 Project: Kafka
  Issue Type: Sub-task
Reporter: Christo Lolov






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Tom Bentley]

Congratulations Sophie!

On Tue, 2 Aug 2022 at 11:54, Robin Moffatt 
wrote:

> Congrats Sophie, great news!
>
>
> --
>
> Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Tue, 2 Aug 2022 at 00:42, Guozhang Wang  wrote:
>
> > Hi everyone,
> >
> > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > committer since Oct. 2020 and has been contributing to the community
> > consistently, especially around Kafka Streams and Kafka java consumer.
> She
> > has also presented about Kafka Streams at Kafka Summit London this year.
> It
> > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> >
> > Congratulations, Sophie!
> >
> > -- Guozhang Wang, on behalf of Apache Kafka PMC
> >
>

Kafka : Round Robin strategy config is not working for consumers

[Anil Pradhan]

hi there,
I'm not sure whether this is the correct group to post this, but pls
suggest if there is any other channel which can guide me on this.

I understand that kafka consumer group concept enables us to come up with
parallel processing of partitions by the underlying consumers within the
same consumer group, meaning if I spin up multiple consumer objects each
belonging to the same consumer group-id would have load balanced across its
underlying consumer instances, by using various partition assignment
strategy configs which are natively available.

We were trying out with RoundRobin assignment strategy config; created an
application which subscribes with 2 topics ( say topic-1 and topic-2, each
having 10 partitions ) and created 10 consumer objects per topic, so that
aspiration is that we have got 20 consumer objects processing from each of
the partitions (total 20 ) of 2 topics; so far so good when we run one
instance of this application as each consumer is attached to one partition
of a topic.

When we try to spin up another instance of this application, where it again
does the same thing by attaching itself to same consumer-group-id, such
that we will now have total 40 consumer objects overall and they get load
balanced in such a way that 10 consumers from earlier instance of the
application(1) released their partition assignments to this new instance
(2) and seeming to be having an equal distribution among both these
instances where each is sharing half of the load ( 10 consumers from each
instance processing messages from 10 partitions each and 10 consumers in
each instance staying idle)

Now the fun starts when we’re trying to spin up 3rd instance of the
application, we expected the load balance to be something like 7, 7 and 6
for each application instance, however it is turning out to be 7, 8 and 5
and sometimes even random allocations, which I don’t consider as fair equal
distribution.

We understand that we can not have more consumers than the partitions
available out there, were looking to have some fine load balancing across
the application instances sharing the same consumer group id to not to
overburden one particular instance as such.

Are we missing any config here or some fundamental understanding, please
guide, many thanks!

Recent commit causes Gradle to load significantly slowly?

[Matthew Benedict de Detrich]

After syncing my fork with upstream Kafka at some point in time in the past I 
noticed that Gradle takes significantly longer to load, i.e. spending 5+ 
minutes in the configuration phase when the Gradle runner starts.

Is anyone else experiencing this problem?

--
Matthew de Detrich
Aiven Deutschland GmbH
Immanuelkirchstraße 26, 10405 Berlin
Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
m: +491603708037
w: aiven.io e: matthew.dedetr...@aiven.io

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Robin Moffatt]

Congrats Sophie, great news!


-- 

Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff


On Tue, 2 Aug 2022 at 00:42, Guozhang Wang  wrote:

> Hi everyone,
>
> I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> committer since Oct. 2020 and has been contributing to the community
> consistently, especially around Kafka Streams and Kafka java consumer. She
> has also presented about Kafka Streams at Kafka Summit London this year. It
> is my pleasure to announce that Sophie agreed to join the Kafka PMC.
>
> Congratulations, Sophie!
>
> -- Guozhang Wang, on behalf of Apache Kafka PMC
>

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi José,

Thanks for having a look at this KIP and thanks for pointing this out, I've had 
a look at KIP-856.

It's good to see there's some overlap in our proposals. we're both proposing:

- Identifying log directories with a UUID
- Extending the storage tool to ensure each log directory has a UUID assigned
- Expanding the topic partition identity to include the log directory UUID

There were differences in our proposals as to how the UUID is to be persisted, 
but I've changed my proposal to match yours — I think adding storage.id to 
meta.properties makes sense.

--
Igor


On Wed, Jul 27, 2022, at 4:42 PM, José Armando García Sancio wrote:
> Hi Igor,
>
> Thanks for the KIP. Looking forward to this improvement. I'll review your KIP.
>
> I should mention that I started a discussion thread on KIP-856: KRaft
> Disk Failure Recovery at
> https://lists.apache.org/thread/ytv0t18cplwwwqcp77h6vry7on378jzj
>
> Both keep introducing similar concepts. For example both KIP introduce
> a storage uuid that is stored in meta.properties. At first glance
> there are some minor differences. I suggest that we review each
> other's KIP so that we can remove these minor differences. What do you
> think?
>
> Thanks!
> --
> -José

Re: [ANNOUNCE] Apache Kafka 3.2.1

[Mickael Maison]

Thanks David for driving this release!

On Tue, Aug 2, 2022 at 1:46 AM David Arthur  wrote:
>
> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.2.1
>
> This is a bugfix release with several fixes since the release of
> 3.2.0. A few of the major issues include:
>
> * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck
>
>
> All of the changes in this release can be found in the release notes:
>
> https://www.apache.org/dist/kafka/3.2.1/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
>
> https://kafka.apache.org/downloads#3.2.1
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream of
> records to one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming
> the input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide,
> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> Zalando, among others.
>
> A big thank you for the following 19 contributors to this release!
>
> Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David
> Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang, Ismael Juma,
> James Hughes, Jason Gustafson, Kirk True, Lucas Bradstreet, Luke Chen,
> Nicolas Guyomar, Niket Goel, Okada Haruki, Shawn Wang, Viktor
> Somogyi-Vass, Walker Carlson
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
>
> Thank you!
>
> Regards,
> David Arthur

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Mickael Maison]

Congratulations Sophie!

On Tue, Aug 2, 2022 at 8:52 AM Josep Prat  wrote:
>
> Congrats Sophie!
>
> On Tue, Aug 2, 2022 at 5:36 AM pengyh  wrote:
>
> >
> > congrats!
> >
> > > Congrats Sophie! 
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Manager, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Immanuelkirchstraße 26, 10405 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B