Re: [ANNOUNCE] New committer: Tom Bentley

[Federico Valeri]

Congrats, Tom!

Well deserved.

On Mon, Mar 15, 2021, 8:09 PM Paolo Patierno  wrote:

> Congratulations Tom!
>
> Get Outlook for Android
>
> 
> From: Guozhang Wang 
> Sent: Monday, March 15, 2021 8:02:44 PM
> To: dev 
> Subject: Re: [ANNOUNCE] New committer: Tom Bentley
>
> Congratulations Tom!
>
> Guozhang
>
> On Mon, Mar 15, 2021 at 11:25 AM Bill Bejeck 
> wrote:
>
> > Congratulations, Tom!
> >
> > -Bill
> >
> > On Mon, Mar 15, 2021 at 2:08 PM Bruno Cadonna  >
> > wrote:
> >
> > > Congrats, Tom!
> > >
> > > Best,
> > > Bruno
> > >
> > > On 15.03.21 18:59, Mickael Maison wrote:
> > > > Hi all,
> > > >
> > > > The PMC for Apache Kafka has invited Tom Bentley as a committer, and
> > > > we are excited to announce that he accepted!
> > > >
> > > > Tom first contributed to Apache Kafka in June 2017 and has been
> > > > actively contributing since February 2020.
> > > > He has accumulated 52 commits and worked on a number of KIPs. Here
> are
> > > > some of the most significant ones:
> > > > KIP-183: Change PreferredReplicaLeaderElectionCommand to use
> > > AdminClient
> > > > KIP-195: AdminClient.createPartitions
> > > > KIP-585: Filter and Conditional SMTs
> > > > KIP-621: Deprecate and replace DescribeLogDirsResult.all() and
> > > .values()
> > > > KIP-707: The future of KafkaFuture (still in discussion)
> > > >
> > > > In addition, he is very active on the mailing list and has helped
> > > > review many KIPs.
> > > >
> > > > Congratulations Tom and thanks for all the contributions!
> > > >
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] 3.0.1 RC0

[Federico Valeri]

Hi Mickael,

- Executed kafka-verify-rc with no issues
- Used the staged Scala 2.13 binaries to run a MM2 DR scenario
- Used staging Maven repository with my own test Java clients

+1 (non-binding)

Thanks
Fede



On Wed, Mar 9, 2022 at 6:01 PM Bill Bejeck  wrote:
>
> Hi Mickael,
>
> Thanks for running the release.
>
> I did the following steps to validate the release:
>
>1. Validated signatures and checksums
>2. Built from source and ran the unit tests
>3. I ran the quickstart steps for ZK and Kafka Streams
>4. Spot checked the docs and Javadocs
>
> I notice the same issue as David regarding referencing the 3.0.0 releases.
> I agree with him that we don't need to block the release and update the
> docs separately.
>
> +1(binding)
>
> Regards,
> Bill
>
> On Wed, Mar 9, 2022 at 10:48 AM David Jacot 
> wrote:
>
> > Thanks for running the release, Mickael.
> >
> > I performed the following validations:
> > * Verified all checksums and signatures.
> > * Built from source and ran unit tests.
> > * Ran the first quickstart steps for both ZK and KRaft.
> > * Spotchecked the Javadocs.
> >
> > However, the document still references 3.0.0 in all places. It seems
> > that it has not been updated for 3.0.1 yet. At least, I don't see any
> > commits for that. I would not block the release on this because we
> > can update the documentation independently.
> >
> > +1 (binding)
> >
> > Best,
> > David
> >
> > On Wed, Mar 9, 2022 at 11:32 AM Tom Bentley  wrote:
> > >
> > > Hi Mickael,
> > >
> > > In addition to the results others have posted I've also validated the
> > > checksums and keys, built from source and executed the unit and
> > integration
> > > tests.
> > >
> > > Overall I'm +1 (binding).
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> > > On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:
> > >
> > > > Hello,
> > > >
> > > >  executed https://github.com/tombentley/kafka-verify-rc - with no
> > issues.
> > > > All checks have passed.
> > > > * checksums, keys
> > > > * unit tests
> > > > * executed few system tests - all passed
> > > >
> > > > +1 (non-binding).
> > > >
> > > > Thank you
> > > >
> > > >
> > > > ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for running the release!
> > > > >
> > > > > I did the following:
> > > > >1. Validated the scala 2.13 checksums
> > > > >2. Spot checked the java docs
> > > > >3. Ran the quick start with scala 2.13 (found a minor bug
> > KAFKA-13718
> > > > > , won't block the
> > > > > release)
> > > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Thank you.
> > > > >
> > > > > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Here is a successful Jenkins build for the 3.0 branch:
> > > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > > > > >
> > > > > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz 
> > wrote:
> > > > > > >
> > > > > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> > > > staging
> > > > > > > Maven repository to run my tests. All seems to work fine, no
> > issues
> > > > > > found.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison <
> > mimai...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > >
> > > > > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > > > > >
> > > > > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> > > > fixed
> > > > > > > > since 3.0.0.
> > > > > > > >
> > > > > > > > Release notes for the 3.0.1 release:
> > > > > > > >
> > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > > > > >
> > > > > > > > *** Please download, test and vote by Thursday, March 10, 6pm
> > GMT
> > > > ***
> > > > > > > >
> > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > >
> > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > > > > >
> > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > >
> > > > > > > > * Javadoc:
> > > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > > > > >
> > > > > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > > > > >
> > > > > > > > * Documentation:
> > > > > > > > https://kafka.apache.org/30/documentation.html
> > > > > > > >
> > > > > > > > * Protocol:
> > > > > > > > https://kafka.apache.org/30/protocol.html
> > > > > > > >
> > > > > > > > * 

Re: New contributor

[Federico Valeri]

Thanks Mickael, now you should have the Confluence account with the
same username (fvaleri).

On Thu, Mar 10, 2022 at 6:23 PM Mickael Maison  wrote:
>
> Hi Federico,
>
> I've added you to the contributor list on JIRA but I couldn't find
> your user in Confluence. Can you verify the username of your
> Confluence account?
>
> Thanks,
> Mickael
>
>
>
> On Thu, Mar 10, 2022 at 5:29 PM Federico Valeri  wrote:
> >
> > Hello, could you add me to the contributor list?
> >
> > Jira/Wiki user: fvaleri
> >
> > Br
> > Fede

New contributor

[Federico Valeri]

Hello, could you add me to the contributor list?

Jira/Wiki user: fvaleri

Br
Fede

Re: [VOTE] KIP-813 Shared State Stores

[Federico Valeri]

Hi, thanks for the KIP.

+1

Just a minor note:

In the rejected alternatives I read "If B would like to lookup s, there are
two possible approaches to take", but then you describe only one.




On Tue, Apr 5, 2022, 6:17 PM Bill Bejeck  wrote:

> Thanks for the KIP, Daan.
>
> I've caught up on the discussion thread and I've gone over the KIP.  This
> seems like a good addition to me.
>
> +1 (binding)
>
> Thanks,
> Bill
>
> On Fri, Apr 1, 2022 at 2:13 PM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> >
> > On 4/1/22 6:47 AM, John Roesler wrote:
> > > Thanks for the KIP, Daan!
> > >
> > > I’m +1 (binding)
> > >
> > > -John
> > >
> > > On Tue, Mar 29, 2022, at 06:01, Daan Gertis wrote:
> > >> I would like to start a vote on this one:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-813%3A+Shareable+State+Stores
> > >>
> > >> Cheers,
> > >> D.
> >
>

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

[Federico Valeri]

Hi Chris, thanks. This looks like a useful feature.

Due to the idempotent nature of PUT, I guess that the last_modified
timestamp won't change if the same request is repeated successively.
Should we add a unit test for that?

On Mon, Sep 4, 2023 at 6:17 AM Ashwin  wrote:
>
> Hi Chris,
>
> Thanks for thinking about this useful feature !
> I had a question regarding
>
> > Since cluster metadata is not required to handle these types of request,
> they will not be forwarded to the leader
>
> And later, we also mention about supporting more scope types in the future.
> Don't you foresee a future scope type which may require cluster metadata ?
> In that case, isn't it better to forward the requests to the leader in the
> initial implementation ?
>
> I would also recommend an additional system test for Standalone herder to
> ensure that the new scope parameter is honored and the response contains
> the last modified time.
>
> Thanks,
> Ashwin
>
> On Sat, Sep 2, 2023 at 5:12 AM Chris Egerton 
> wrote:
>
> > Hi all,
> >
> > Can't imagine a worse time to publish a new KIP (it's late on a Friday and
> > we're in the middle of the 3.6.0 release), but I wanted to put forth
> > KIP-976 for discussion:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976%3A+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect
> >
> > TL;DR: The API to dynamically adjust log levels at runtime with Connect is
> > great, and I'd like to augment it with support to adjust log levels for
> > every worker in the cluster (instead of just the worker that receives the
> > REST request).
> >
> > I look forward to everyone's thoughts, but expect that this will probably
> > take a bump or two once the dust has settled on 3.6.0. Huge thanks to
> > everyone that's contributed to that release so far, especially our release
> > manager Satish!
> >
> > Cheers,
> >
> > Chris
> >

Re: Apache Kafka 3.6.0 release

[Federico Valeri]

Hi Satish, this is a small documentation fix about ZK to KRaft
migration, that we would like to backport to 3.5 and 3.6 branches. Are
you ok with that?

https://github.com/apache/kafka/pull/14366

On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana  wrote:
>
> Thanks David for the quick resolution.
>
> ~Satish.
>
> On Tue, 12 Sept 2023 at 22:51, David Arthur
>  wrote:
> >
> > Satish,
> >
> > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> >
> > Thanks!
> > David
> >
> > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma  wrote:
> >
> > > Justine,
> > >
> > > Probably best to have the conversation in the JIRA ticket vs the release
> > > thread. Generally, we want to only include low risk bug fixes that are
> > > fully compatible in patch releases.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Thanks Satish. I understand.
> > > > Just curious, is this something that could be added to 3.6.1? It would 
> > > > be
> > > > nice to say that hanging transactions are fully covered in a 3.6 
> > > > release.
> > > > I'm not as familiar with the rules around minor releases, but adding it
> > > > there would give more time to ensure stability.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana  > > >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > > regression and it involves changes to the API implementation. Let us
> > > > > plan to add the gap in the release notes as you mentioned.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > We just discovered a gap in KIP-890 part 1. We currently don't 
> > > > > > verify
> > > > on
> > > > > > txn offset commits, so it is still possible to have hanging
> > > > transactions
> > > > > on
> > > > > > the consumer offsets partitions.
> > > > > > I've opened a jira to wire the verification in that request.
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > > >
> > > > > > This also isn't a regression, but it would be nice to have part 1
> > > fully
> > > > > > complete. I have opened a PR with the fix:
> > > > > > https://github.com/apache/kafka/pull/14370.
> > > > > >
> > > > > > I understand if there are concerns about last minute changes to this
> > > > API
> > > > > > and we can hold off if that makes the most sense.
> > > > > > If we take that route, I think we should still keep verification for
> > > > the
> > > > > > data partitions since it still provides full protection there and
> > > > > improves
> > > > > > the transactions experience. We will need to call out the gap in the
> > > > > > release notes for consumer offsets partitions
> > > > > >
> > > > > > Let me know what you think.
> > > > > > Justine
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur
> > > > > >  wrote:
> > > > > >
> > > > > > > Another (small) ZK migration issue was identified. This one isn't 
> > > > > > > a
> > > > > > > regression (it has existed since 3.4), but I think it's reasonable
> > > to
> > > > > > > include. It's a small configuration check that could potentially
> > > save
> > > > > end
> > > > > > > users from some headaches down the line.
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-15450
> > > > > > > https://github.com/apache/kafka/pull/14367
> > > > > > >
> > > > > > > I think we can get this one committed to trunk today.
> > > > > > >
> > > > > > > -David
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma 
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > That sounds great. I think we should aim to only allow blockers
> > > > > > > > (regressions, impactful security issues, etc.) on the 3.6 branch
> > > > > until
> > > > > > > > 3.6.0 is out.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Sep 9, 2023, 12:20 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Ismael,
> > > > > > > > > It looks like we will publish RC0 by 14th Sep.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Satish.
> > > > > > > > >
> > > > > > > > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma 
> > > > > > > > > 
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Satish,
> > > > > > > > > >
> > > > > > > > > > Do you have a sense of when we'll publish RC0?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur
> > > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > > Quick update on my two blockers: KAFKA-15435 is merged to
> > > > > trunk and
> > > > > > > > > > > 

Re: Complete Kafka replication protocol description

[Federico Valeri]

Hi Jack, thanks for providing this material. I'll spend some time
digging into it.

On Mon, Sep 11, 2023 at 6:56 PM Jack Vanlightly  wrote:
>
> Hi all,
>
> I agree that we should have the protocol description and specification in
> the Kafka repository. I have two other specifications to contribute
> including the KRaft implementation of Raft and the next gen consumer group
> protocol (WIP). I also have a formal prose description of the Kafka
> replication protocol for how it works today. That should appear under the
> 3.5 directory sometime this week.
>
> Regarding moving this kind of stuff into the Kafka repo, there are a number
> of additional topics such as:
>
>- Where to place formal prose descriptions and specifications of
>protocols in the Kafka repo.

I'm thinking about a dedicated folder (i.e. tlaplus) with a main
README.md where we briefly explain what's contained.

>- Given we may add more specifications and potentially other formal
>prose descriptions, how to structure things?

Each spec would live in its own sub folder under specs, along with a
README.md containing the prose description. I think we should also
provide the configuration files, and a bash script to easily run the
model checker on them. Wdyt?

>- Do we use markdown like I have done here with svg files (created in
>Excalidraw with the Excalidraw bits embedded in the svg)., or do we use
>LaTex? Some people may wish to read this directly on GitHub and perhaps
>others like a PDF. Markdown is more accessible and perhaps more likely to
>fulfil the living document approach.

I agree that Markdown is more convenient and will increase the chance
that these specifications will receive updates.

>
> Thanks
> Jack
>
>
> On Mon, Sep 11, 2023 at 1:45 PM Divij Vaidya 
> wrote:
>
> > This is very useful Jack!
> >
> > 1. We are missing a replication protocol specification in our project.
> > Ideally it should be a living document and adding what you wrote to
> > the docs/design would be a great start towards that goal.
> > 2. I have also been building on top of some existing TLA+ to add
> > changes to replication protocol brought by features such as Tiered
> > Storage, local retention etc. at
> >
> > https://github.com/divijvaidya/kafka-specification/blob/master/KafkaReplication.tla
> > 3. Apart from verifying correctness, I believe a TLA+ model will also
> > help developers quickly iterate through their fundamental assumptions
> > while making changes. As an example, we recently had a discussion in
> > the community on whether to increase leader epoch with shrink/expand
> > ISR or not. There was another discussion on whether we can choose the
> > replica with the largest end offset as the new leader to reduce
> > truncation. In both these cases, we could have quickly modified the
> > existing TLA+ model, run through it and verify that the assumptions
> > still hold true. It would be great if we can take such discussions as
> > an example and demonstrate how TLA+ could have benefitted the
> > community. It would help make the case for adding the TLA+ spec as
> > part of the community owned project itself.
> >
> > Right now, things are a bit busy on my end, but I am looking forward
> > to exploring what you shared above in the coming weeks (perhaps a
> > first review by end of sept).
> >
> > Thank you again for starting this conversation.
> >
> > --
> > Divij Vaidya
> >
> > On Mon, Sep 11, 2023 at 4:49 AM Haruki Okada  wrote:
> > >
> > > Hi Jack,
> > >
> > > Thank you for the great work, not only the spec but also for the
> > > comprehensive documentation about the replication.
> > > Actually I wrote some TLA+ spec to verify unclean leader election
> > behavior
> > > before so I will double-check my understanding with your complete spec :)
> > >
> > >
> > > Thanks,
> > >
> > > 2023年9月10日(日) 21:42 David Jacot :
> > >
> > > > Hi Jack,
> > > >
> > > > This is great! Thanks for doing it. I will look into it when I have a
> > bit
> > > > of time, likely after Current.
> > > >
> > > > Would you be interested in contributing it to the main repository? That
> > > > would be a great contribution to the project. Having it there would
> > allow
> > > > the community to maintain it while changes to the protocol are made.
> > That
> > > > would also pave the way for having other specs in the future (e.g. new
> > > > consumer group protocol).
> > > >
> > > > Best,
> > > > David
> > > >
> > > > Le dim. 10 sept. 2023 à 12:45, Jack Vanlightly 
> > a
> > > > écrit :
> > > >
> > > > > Hi all,
> > > > >
> > > > > As part of my work on formally verifying different parts of Apache
> > Kafka
> > > > > and working on KIP-966 I have built up a lot of knowledge about how
> > the
> > > > > replication protocol works. Currently it is mostly documented across
> > > > > various KIPs and in the code itself. I have written a complete
> > protocol
> > > > > description (with KIP-966 changes applied) which is inspired by the
> > > > precise
> > 

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Federico Valeri]

Hi Krishna, thanks for opening this discussion.

I see you created two separate KIPs (974 and 975), but there are some
common points (build system and test plan).

Currently, the Docker image used for system tests is only supported in
that limited scope, so the maintenance burden is minimal. Providing
official Kafka images would be much more complicated. Have you
considered how the image rebuild process would work in case a high
severity CVE comes out for a non Kafka image dependency? In that case,
there will be no Kafka release.

Br
Fede

On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
 wrote:
>
> Hi,
> I want to submit a KIP to deliver an experimental Apache Kafka docker image.
> The proposed docker image can launch brokers with sub-second startup time
> and minimal memory footprint by leveraging a GraalVM based native Kafka
> binary.
>
> KIP-974: Docker Image for GraalVM based Native Kafka Broker
> 
>
> Regards,
> Krishna

Re: [VOTE] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

[Federico Valeri]

Hi Jakub, thanks for this KIP.

+1 (non binding)

Thanks
Fede

On Wed, Oct 25, 2023 at 4:45 PM Manikumar  wrote:
>
> Hi,
>
> Thanks for the KIP.
>
> +1 (binding)
>
>
> Thanks.
>
> On Wed, Oct 25, 2023 at 1:37 AM Jakub Scholz  wrote:
>
> > Hi all,
> >
> > I would like to start a vote for the KIP-978: Allow dynamic reloading of
> > certificates with different DN / SANs
> > <
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128
> > >
> > .
> >
> > Thanks & Regards
> > Jakub
> >

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Federico Valeri]

Hi Krishna, thanks for updating the KIP and all the work you are
putting into that.

The release process LGTM. In the other KIP I see that there will be
some automation for building, testing and scanning for CVEs. Is this
also true for native images?

I see you are proposing to use Alpine as the base image. I would add
Distroless to the rejected alternatives with the motivation. Maybe we
can do the same for the GraalVM distribution of choice.

On Fri, Oct 20, 2023 at 12:02 PM Manikumar  wrote:
>
> Hi,
>
> > For the native AK docker image, we are considering '*kafka-local*' as it
> clearly signifies that this image is intended exclusively for local
>
> I am not sure, if there is any naming pattern for graalvm based images. Can
> we include "graalvm" to the image name like "kafka-graalvm-native".
> This will clearly indicate this is graalvm based image.
>
>
> Thanks. Regards
>
>
>
>
> On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Federico,
> > Thanks for the feedback and apologies for the delay.
> >
> > I've included a section in the KIP on the release process. I would greatly
> > appreciate your insights after reviewing it.
> >
> > Regards,
> > Krishna
> >
> > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri 
> > wrote:
> >
> > > Hi Krishna, thanks for opening this discussion.
> > >
> > > I see you created two separate KIPs (974 and 975), but there are some
> > > common points (build system and test plan).
> > >
> > > Currently, the Docker image used for system tests is only supported in
> > > that limited scope, so the maintenance burden is minimal. Providing
> > > official Kafka images would be much more complicated. Have you
> > > considered how the image rebuild process would work in case a high
> > > severity CVE comes out for a non Kafka image dependency? In that case,
> > > there will be no Kafka release.
> > >
> > > Br
> > > Fede
> > >
> > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > >  wrote:
> > > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >
> > > >
> > > > Regards,
> > > > Krishna
> > >
> >

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Federico Valeri]

Hi Krishna, thanks for the updates, LGTM.

On Mon, Oct 30, 2023 at 3:36 PM Krishna Agarwal
 wrote:
>
> Hi Federico,
> Thanks for the feedback.
>
>1. Yes, we will add the building, testing and scanning automation for
>this Docker Image along with the flow mentioned in KIP-975. (Updated in the
>KIP)
>2. Added the other alternatives to the "rejected alternatives" section,
>instead of the main sections. (Updated in the KIP)
>3. Regarding the release process- In the KIP-975, it was concluded that
>there shouldn't be any docker specific release process. If there is a high
>severity CVE, we should release a new version of Kafka for the affected
>branch. It would include the latest Kafka code from the branch. In my
>opinion we should keep the same release process here for consistency.
>(Updated in the KIP)
>KIP-975 Release Process:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-ReleaseProcess
>Discussion thread for the same:
>https://lists.apache.org/thread/05t8ccvhp3fotfftgm7dzn8wobkl59l4
>
>
> Regards,
> Krishna
>
> On Wed, Oct 25, 2023 at 9:50 PM Federico Valeri 
> wrote:
>
> > Hi Krishna, thanks for updating the KIP and all the work you are
> > putting into that.
> >
> > The release process LGTM. In the other KIP I see that there will be
> > some automation for building, testing and scanning for CVEs. Is this
> > also true for native images?
> >
> > I see you are proposing to use Alpine as the base image. I would add
> > Distroless to the rejected alternatives with the motivation. Maybe we
> > can do the same for the GraalVM distribution of choice.
> >
> > On Fri, Oct 20, 2023 at 12:02 PM Manikumar 
> > wrote:
> > >
> > > Hi,
> > >
> > > > For the native AK docker image, we are considering '*kafka-local*' as
> > it
> > > clearly signifies that this image is intended exclusively for local
> > >
> > > I am not sure, if there is any naming pattern for graalvm based images.
> > Can
> > > we include "graalvm" to the image name like "kafka-graalvm-native".
> > > This will clearly indicate this is graalvm based image.
> > >
> > >
> > > Thanks. Regards
> > >
> > >
> > >
> > >
> > > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > > > Hi Federico,
> > > > Thanks for the feedback and apologies for the delay.
> > > >
> > > > I've included a section in the KIP on the release process. I would
> > greatly
> > > > appreciate your insights after reviewing it.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Krishna, thanks for opening this discussion.
> > > > >
> > > > > I see you created two separate KIPs (974 and 975), but there are some
> > > > > common points (build system and test plan).
> > > > >
> > > > > Currently, the Docker image used for system tests is only supported
> > in
> > > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > > official Kafka images would be much more complicated. Have you
> > > > > considered how the image rebuild process would work in case a high
> > > > > severity CVE comes out for a non Kafka image dependency? In that
> > case,
> > > > > there will be no Kafka release.
> > > > >
> > > > > Br
> > > > > Fede
> > > > >
> > > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > >  wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > docker
> > > > > image.
> > > > > > The proposed docker image can launch brokers with sub-second
> > startup
> > > > time
> > > > > > and minimal memory footprint by leveraging a GraalVM based native
> > Kafka
> > > > > > binary.
> > > > > >
> > > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > > <
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Krishna
> > > > >
> > > >
> >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Federico Valeri]

Hi Satish, I did the following to verify the release:

- Verified signature and checksum
- Built from source with Java 17 and Scala 2.13
- Ran all unit and integration tests
- Spot checked release notes and documentation
- Ran a custom client using staging artifacts on a 3-nodes cluster
- Tested tiered storage with one of the available RSM implementations

+1 (non binding)

Thanks
Fede


On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
>
> Hi Satish,
>
> I verified with:
> 1. Ran quick start in KRaft for scala 2.12 artifact
> 2. Making sure the checksum are correct
> 3. Browsing release notes, documents, javadocs, protocols.
>
> I filed KAFKA-15491 for
> log output improvement while testing stream application.
> It won't be blocker in v3.6.0.
>
> For KAFKA-15489 , I'm
> fine if we decide to fix it in v3.6.1/v3.7.0.
>
> +1 (binding) from me.
>
> Thank you.
> Luke
>
> On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:
>
> > Given that this is not a regression and there have been no reports for over
> > a year, I think it's ok for this to land in 3.6.1.
> >
> > Ismael
> >
> > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Luke for reporting KRaft issue[1].
> > >
> > > I am not sure whether it is a release blocker for 3.6.0. Need input
> > > from other KRaft experts also to finalize the decision. Even if we
> > > adopt a fix, do not we need to bake it for some time before it is
> > > pushed to production to avoid any regressions as this change is in the
> > > critical paths?
> > >
> > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > I found the current KRaft implementation will have "split brain" issue
> > > when
> > > > network partition happens, which will cause inconsistent metadata
> > > returned
> > > > from the controller.
> > > > Filed KAFKA-15489 
> > > for
> > > > this issue, and PR  is
> > ready
> > > > for review.
> > > >
> > > > Even though this is not a regression issue (this has already existed
> > > since
> > > > the 1st release of KRaft feature), I think this is an important issue
> > > since
> > > > KRaft is announced production ready.
> > > > Not sure what other people's thoughts are.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat  > >
> > > > wrote:
> > > >
> > > > > Hi Satish,
> > > > >
> > > > > I ran the following validation steps:
> > > > > - Built from source with Java 11 and Scala 2.13
> > > > > - Verified Signatures and hashes of the artifacts generated
> > > > > - Navigated through Javadoc including links to JDK classes
> > > > > - Run the unit tests
> > > > > - Run integration tests
> > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > >
> > > > >
> > > > > I +1 this release (non-binding)
> > > > >
> > > > > Thanks for your efforts!
> > > > >
> > > > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Greg for verifying the release including the earlier
> > > > > > blocker(KAFKA-15473) verification.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
> > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I verified the functionality of KIP-898 and the recent fix for
> > > > > > > KAFKA-15473 with the following steps:
> > > > > > >
> > > > > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > > > > publicly available plugins installed
> > > > > > > 2. I captured the output of /connector-plugins
> > > > > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > > > > 4. I captured the output of /connector-plugins with various
> > > settings
> > > > > > > of plugin.discovery
> > > > > > > 5. I ran the migration script to add manifests to my plugins
> > > > > > > 6. I captured the output of /connector-plugins with various
> > > settings
> > > > > > > of plugin.discovery
> > > > > > > 7. I downgraded the worker to 3.5.1
> > > > > > > 8. I diffed the output of /connector-plugins across the different
> > > > > > > cases and observed the expected changes.
> > > > > > > a. When plugins are migrated for 3.6.0, all modes produce
> > > identical
> > > > > > > results.
> > > > > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > > > > service_load is missing plugins
> > > > > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > > > > constructors are hidden, AK plugins now have versions,
> > > multi-interface
> > > > > > > plugins now show each interface 

Re: [VOTE] 3.6.0 RC2

[Federico Valeri]

Hi Satish, I did the following to verify the release:

- Built from source with Java 17 and Scala 2.13
- Ran all unit and integration tests
- Spot checked documentation
- Ran custom client applications using staging artifacts on a 3-nodes cluster
- Tested tiered storage with one of the available RSM implementations

+1 (non binding)

Thanks
Fede

On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
>
> Hi Satish,
>
> I verified with:
> 1. Ran quick start in KRaft for scala 2.12 artifact
> 2. Making sure the checksum are correct
> 3. Browsing release notes, documents, javadocs, protocols.
> 4. Verified the tiered storage feature works well.
>
> +1 (binding).
>
> Thanks.
> Luke
>
>
>
> On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:
>
> > +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven
> > artifacts and run my tests. Everything seems to work fine for me.
> >
> > Thanks
> > Jakub
> >
> > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana 
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for the release of Apache Kafka 3.6.0.
> > > Some of the major features include:
> > >
> > > * KIP-405 : Kafka Tiered Storage
> > > * KIP-868 : KRaft Metadata Transactions
> > > * KIP-875: First-class offsets support in Kafka Connect
> > > * KIP-898: Modernize Connect plugin discovery
> > > * KIP-938: Add more metrics for measuring KRaft performance
> > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > * KIP-917: Additional custom metadata for remote log segment
> > >
> > > Release notes for the 3.6.0 release:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > There are a few runs of unit/integration tests. You can see the latest
> > > at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > > continue running a few more iterations.
> > > System tests:
> > > We will send an update once we have the results.
> > >
> > > Thanks,
> > > Satish.
> > >
> >

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

[Federico Valeri]

Hi Hailey, thanks for the KIP.

I also agree that the two mutually exclusive args are better. In order
to be consistent with the other tools, I would suggest to use
--process-role and --node-id (hyphen instead of dot). Can you also
update the KIP?

On Mon, Oct 2, 2023 at 10:18 PM Hailey Ni  wrote:
>
> Hi Kamal,
>
> I think the broker.id property has been replaced with the `node.id` property
> in KRaft.  The documentation for `node.id` says it is required (
> https://github.com/apache/kafka/blob/72e275f6ea867747e6b4e524c80d5ebd726ac25b/core/src/main/scala/kafka/server/KafkaConfig.scala#L741),
> and the QuickStart files all use it (
> https://github.com/apache/kafka/tree/72e275f6ea867747e6b4e524c80d5ebd726ac25b/config/kraft).
> It is technically true that these two configs are treated as synonyms of
> one another (
> https://github.com/apache/kafka/blob/72e275f6ea867747e6b4e524c80d5ebd726ac25b/core/src/main/scala/kafka/server/KafkaConfig.scala#L1587-L1597),
> so if you specify either one the process will still recognize it and
> start.  But it makes sense to exclusively use `node.id` in KRaft because a
> node isn't necessarily a broker anymore; it could be a controller (or even
> a combined broker+controller).
>
> Thanks,
> Hailey
>
> On Mon, Oct 2, 2023 at 1:17 PM Hailey Ni  wrote:
>
> > Hi Ismeal,
> >
> > Thanks for the comments. I'll change the implementation to use a pair of
> > mutually exclusive args --process.roles and --node.id.
> >
> > Thanks,
> > Hailey
> >
> > On Mon, Oct 2, 2023 at 6:34 AM Ismael Juma  wrote:
> >
> >> Hi Ron,
> >>
> >> Yes, that's what I am proposing, yes.
> >>
> >> Ismael
> >>
> >> On Sat, Sep 30, 2023 at 2:30 PM Ron Dagostino  wrote:
> >>
> >> > Thanks, Ismael.  I think you are proposing a pair of mutually exclusive
> >> > args --process.roles and --node.id, right?  I agree that is more
> >> > user-friendly than the --required-config arg, and it comes at the
> >> possible
> >> > expense of generality.  So that’s the tradeoff between the two, I think.
> >> > No other config comes to mind now that we’ve identified these two.  I
> >> think
> >> > the two specific and mutually exclusive parameters would be the way to
> >> go
> >> > unless someone else identifies still more options that people might
> >> want.
> >> >
> >> > Did I get that right, or were you proposing something different?
> >> >
> >> > Ron
> >> >
> >> > > On Sep 30, 2023, at 10:42 AM, Ismael Juma  wrote:
> >> > >
> >> > > Hi,
> >> > >
> >> > > Thanks for the KIP. I think this approach based on configs is a bit
> >> too
> >> > > open ended and not very user friendly. Why don't we simply provide
> >> flags
> >> > > for the things a user may care about? So far, it seems like we have
> >> two
> >> > > good candidates (node id and process role). Are there any others?
> >> > >
> >> > > Ismael
> >> > >
> >> > >> On Fri, Sep 29, 2023 at 6:19 PM Hailey Ni 
> >> > wrote:
> >> > >>
> >> > >> Hi Ron,
> >> > >>
> >> > >> I think you made a great point, making the "name" arbitrary instead
> >> of
> >> > >> hard-coding it will make the functionality much more flexible. I've
> >> > updated
> >> > >> the KIP and the code accordingly. Thanks for the great idea!
> >> > >>
> >> > >> Thanks,
> >> > >> Hailey
> >> > >>
> >> > >>
> >> > >>> On Fri, Sep 29, 2023 at 2:34 PM Ron Dagostino 
> >> > wrote:
> >> > >>>
> >> > >>> Thanks, Hailey.  Is there a reason to restrict it to just
> >> > >>> process.roles and node.id?  Someone might want to do
> >> > >>> "--required-config any.name=whatever.value", for example, and at
> >> first
> >> > >>> glance I don't see a reason why the implementation should be any
> >> > >>> different -- it seems it would probably be easier to not have to
> >> worry
> >> > >>> about restricting to specific cases, actually.  WDYT?
> >> > >>>
> >> > >>> Ron
> >> > >>>
> >> > >>> On Fri, Sep 29, 2023 at 5:12 PM Hailey Ni  >> >
> >> > >>> wrote:
> >> > 
> >> >  Updated. Please let me know if you have any additional comments.
> >> Thank
> >> > >>> you!
> >> > 
> >> >  On Thu, Sep 21, 2023 at 3:02 PM Hailey Ni 
> >> wrote:
> >> > 
> >> > > Hi Ron. Thanks for the response. I agree with your point. I'll
> >> make
> >> > >> the
> >> > > corresponding changes in the KIP and KAFKA-15471
> >> > > .
> >> > >
> >> > > On Thu, Sep 21, 2023 at 1:40 PM Ron Dagostino 
> >> > >>> wrote:
> >> > >
> >> > >> Hi Hailey.  No, I just looked, and zookeeper-server-stop does not
> >> > >> have
> >> > >> any facility to be specific about which ZK nodes to signal.  So
> >> > >> providing the ability in kafka-server-stop to be more specific
> >> than
> >> > >> just "signal all controllers" or "signal all brokers" would be a
> >> > >> bonus
> >> > >> and therefore not necessarily required.  But if it is easy to
> >> > >> achieve
> >> > >> and doesn't add any additional cognitive load -- and at first
> >> glance
> >> > >> it 

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

[Federico Valeri]

Hi Chris,

+1 (non binding)

Thanks
Fede

On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya  wrote:
>
> Hi Chris,
>
> Thanks for the KIP!
> +1 (binding)
>
> Yash
>
> On Fri, Oct 6, 2023 at 9:54 PM Greg Harris 
> wrote:
>
> > Hey Chris,
> >
> > Thanks for the KIP!
> > I think that preserving the ephemeral nature of the logging change is
> > the right choice here, and using the config topic for intra-cluster
> > broadcast is better than REST forwarding.
> >
> > +1 (binding)
> >
> > Thanks,
> > Greg
> >
> > On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-976, which augments the existing
> > dynamic
> > > logger adjustment REST API for Kafka Connect to apply changes
> > cluster-wide
> > > instead on a per-worker basis.
> > >
> > > The KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect
> > >
> > > The discussion thread:
> > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5
> > >
> > > Cheers,
> > >
> > > Chris
> >

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

[Federico Valeri]

+1 (non binding)

Thanks.

On Mon, Oct 23, 2023 at 9:48 AM Kamal Chandraprakash
 wrote:
>
> +1 (non-binding). Thanks for the KIP!
>
> On Mon, Oct 23, 2023, 12:55 Hailey Ni  wrote:
>
> > Hi all,
> >
> > I'd like to call a vote on KIP-979 that will allow users to independently
> > stop KRaft processes.
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-979%3A+Allow+independently+stop+KRaft+processes
> >
> > Thanks,
> > Hailey
> >

Re: [DISCUSS] KIP-830: Allow disabling JMX Reporter

[Federico Valeri]

Hi Mickael, what about setting the default value to JmxReporter while
also maintaining the old behavior? When a user is setting
metric.reporters without explicitly including the JmxReporter, we
could simply print a warning message which says "The automatic
addition of the JmxReporter is deprecated. Add it explicitly to the
metric.reporters list if needed.", or something like that. That way we
can avoid adding a new property, and drop the implicit setting in the
next major release.

On Wed, Apr 13, 2022 at 11:49 AM Mickael Maison
 wrote:
>
> Hi,
>
> I've created a small KIP to allow disabling JMXreporter:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-830%3A+Allow+disabling+JMX+Reporter
>
> Let me know if you have any feedback.
>
> Thanks,
> Mickael

Re: [VOTE] KIP-830: Allow disabling JMX Reporter

[Federico Valeri]

+1 (non binding)

Thanks.

On Tue, May 17, 2022 at 3:47 PM Mickael Maison  wrote:
>
> Hi,
>
> I'd like to start a vote on KIP-830 which proposes a method for a
> method for disabling JMXReporter and making JMXReporter behave like
> other reporters in the next major release when it will need to be
> explicitly listed in metric.reporters to be enabled.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-830%3A+Allow+disabling+JMX+Reporter
>
> Let me know if you have feedback,
>
> Thanks,
> Mickael

Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API

[Federico Valeri]

Thanks Mickael.

+1 (non binding)

On Wed, May 18, 2022 at 11:08 AM Divij Vaidya  wrote:
>
> +1 non binding.
>
> Divij Vaidya
>
>
>
> On Tue, May 17, 2022 at 6:16 PM Igor Soarez  wrote:
>
> > Thanks for this KIP Mickael.
> >
> > +1 non binding
> >
> > --
> > Igor
> >
> > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote:
> > > Hi Mickael,
> > >
> > > +1 (binding) from me.
> > > Thanks for the KIP!
> > >
> > > Luke
> > >
> > > On Tue, May 17, 2022 at 9:30 PM Mickael Maison  > >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd like to start a vote on KIP-827. It proposes exposing the total
> > >> and usable space of logdirs
> > >> via the DescribeLogDirs API:
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> >

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

[Federico Valeri]

Hi Bruno,

+1 (non binding)

- Checked documentation and blog post
- Ran some client tests using the staged artifacts
- Verified the signing key and sha512 sum
- Built from source
- Ran unit and integration tests

Thanks
Fede

On Thu, May 12, 2022 at 12:03 PM Bruno Cadonna  wrote:
>
> Hi all,
>
> Please review the blog post for the Apache Kafka 3.2.0 release:
>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache8
>
> I will accept comments until Monday, May 16th EOD PT.
>
> Best,
> Bruno
>
>
> On 06.05.22 14:52, 'David Jacot' via kafka-clients wrote:
> > Thanks for running the release, Bruno.
> >
> > I performed the following validations:
> > * Verified all checksums and signatures.
> > * Built from source and ran unit tests.
> > * Ran the first quickstart steps for both ZK and KRaft.
> > * Spotchecked the doc and the Javadocs.
> >
> > +1 (binding)
> >
> > Best,
> > David
> >
> > On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:
> >>
> >> +1 (non-binding).
> >>
> >> I used the Scala 2.13 binaries and the staged Maven artifacts and ran
> >> various tests with them. Thanks for doing the release.
> >>
> >> Jakub
> >>
> >> On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna  wrote:
> >>
> >>> Hello Kafka users, developers and client-developers,
> >>>
> >>> This is the second candidate for release of Apache Kafka 3.2.0.
> >>>
> >>> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> >>> * StandardAuthorizer for KRaft (KIP-801)
> >>> * Send a hint to the partition leader to recover the partition (KIP-704)
> >>> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> >>> * kafka-console-producer writes headers and null values (KIP-798 and
> >>> KIP-810)
> >>> * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> >>> * Static membership protocol lets the leader skip assignment (KIP-814)
> >>> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> >>> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> >>> * Connect APIs list all connector plugins and retrieve their
> >>> configuration (KIP-769)
> >>> * TimestampConverter SMT supports different unix time precisions (KIP-808)
> >>> * Connect source tasks handle producer exceptions (KIP-779)
> >>>
> >>>
> >>> Release notes for the 3.2.0 release:
> >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> >>>
> >>> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> >>>
> >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>> https://kafka.apache.org/KEYS
> >>>
> >>> * Release artifacts to be voted upon (source and binary):
> >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> >>>
> >>> * Maven artifacts to be voted upon:
> >>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>>
> >>> * Javadoc:
> >>> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> >>>
> >>> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> >>> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> >>>
> >>> * Documentation:
> >>> https://kafka.apache.org/32/documentation.html
> >>>
> >>> * Protocol:
> >>> https://kafka.apache.org/32/protocol.html
> >>>
> >>> * Successful Jenkins builds for the 3.2 branch:
> >>> Unit/integration tests: I'll share a link once the builds complete
> >>> System tests:
> >>> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> >>>
> >>> /**
> >>>
> >>> Thanks,
> >>> Bruno
> >>>
> >

Re: [Vote] KIP-787 - MM2 Interface to manage Kafka resources

[Federico Valeri]

Hi Omnia, this will be really useful, especially in cloud environment.

+1 non binding

Thanks
Fede

On Tue, May 31, 2022 at 5:28 PM Mickael Maison  wrote:
>
> Hi Omnia,
>
> I think the approach you settled on is the best option, this will
> allow integrating MirrorMaker in more environments.
>
> +1 binding
>
> Thanks for the KIP (and your persistence!)
> Mickael
>
> On Mon, May 30, 2022 at 12:09 PM Omnia Ibrahim  
> wrote:
> >
> > Hi,
> > Can I get a vote on this, please?
> > Thanks
> >
> > On Wed, May 25, 2022 at 11:15 PM Omnia Ibrahim 
> > wrote:
> >
> > > Hi,
> > > I'd like to start a vote on KIP-787
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-787
> > > %3A+MM2+Interface+to+manage+Kafka+resources
> > > 
> > >
> > > Thanks
> > > Omnia
> > >

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Federico Valeri]

Hi Luke, thanks for the KIP.

I think we miss the "dir" key in "remainingLogsToRecover" ObjectName.

kafka.log:type=LogManager,name=remainingLogsToRecover,dir=([-._\/\w\d\s]+)
kafka.log:type=LogManager,name=remainingSegmentsToRecover,dir=([-._\/\w\d\s]+),threadNum=([0-9]+)

Example:

Broker configuration:
log.dirs=/tmp/log1,/tmp/log2
num.recovery.threads.per.data.dir=2

Registered ObjectNames:
kafka.log:type=LogManager,name=remainingLogsToRecover,dir=/tmp/log1
kafka.log:type=LogManager,name=remainingLogsToRecover,dir=/tmp/log2
kafka.log:type=LogManager,name=remainingSegmentsToRecover,dir=/tmp/log1,threadNum=0
kafka.log:type=LogManager,name=remainingSegmentsToRecover,dir=/tmp/log1,threadNum=1
kafka.log:type=LogManager,name=remainingSegmentsToRecover,dir=/tmp/log2,threadNum=0
kafka.log:type=LogManager,name=remainingSegmentsToRecover,dir=/tmp/log2,threadNum=1

On Sat, Jun 4, 2022 at 4:42 AM Luke Chen  wrote:
>
> Hi Jun,
>
> Thanks for the comment.
>
> I've updated the KIP as:
> 1. remainingLogsToRecover*y* -> remainingLogsToRecover
> 2. remainingSegmentsToRecover*y* -> remainingSegmentsToRecover
> 3. The description of remainingSegmentsToRecover: The remaining segments
> for the current log assigned to the recovery thread.
>
> Thank you.
> Luke
>
> On Sat, Jun 4, 2022 at 12:54 AM Jun Rao  wrote:
>
> > Hi, Luke,
> >
> > Thanks for the explanation.
> >
> > 10. It makes sense to me now. Instead of using a longer name, perhaps we
> > could keep the current name, but make the description clear that it's the
> > remaining segments for the current log assigned to a thread. Also, would it
> > be better to use ToRecover instead of ToRecovery?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jun 3, 2022 at 1:18 AM Luke Chen  wrote:
> >
> > > Hi Jun,
> > >
> > > > how do we implement kafka.log
> > >
> > >
> > :type=LogManager,name=remainingSegmentsToRecovery,dir=([-._\/\w\d\s]+),threadNum=([0-9]+),
> > > which tracks at the segment level?
> > >
> > > It looks like the name is misleading.
> > > Suppose we have 2 log recovery threads
> > > (num.recovery.threads.per.data.dir=2),
> > > and 10 logs to iterate through
> > > As mentioned before, we don't know how many segments in each log until
> > the
> > > log is iterated(loaded)
> > > So, when thread-1 iterates logA, it gets the segments to recover, and
> > > expose the number to `remainingSegmentsToRecovery` metric.
> > > And the same for thread-2 iterating logB.
> > > That is, the metric showed in `remainingSegmentsToRecovery` is actually
> > the
> > > remaining segments to recover "in a specific log".
> > >
> > > Maybe I should rename it: remainingSegmentsToRecovery ->
> > > remainingSegmentsToRecoverInCurrentLog
> > > WDYT?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Jun 3, 2022 at 1:27 AM Jun Rao  wrote:
> > >
> > > > Hi, Luke,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 10. You are saying it's difficult to track the number of segments to
> > > > recover. But how do we
> > > > implement
> > > >
> > >
> > kafka.log:type=LogManager,name=remainingSegmentsToRecovery,dir=([-._\/\w\d\s]+),threadNum=([0-9]+),
> > > > which tracks at the segment level?
> > > >
> > > > Jun
> > > >
> > > > On Thu, Jun 2, 2022 at 3:39 AM Luke Chen  wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > Thanks for the comment.
> > > > >
> > > > > Yes, I've tried to work on this way to track the number of remaining
> > > > > segments, but it will change the design in UnifiedLog, so I only
> > track
> > > > the
> > > > > logs number.
> > > > > Currently, we will load all segments and recover those segments if
> > > needed
> > > > > "during creating UnifiedLog instance". And also get the log offsets
> > > here
> > > > > <
> > > > >
> > > >
> > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/UnifiedLog.scala#L1819-L1842
> > > > > >
> > > > > .
> > > > > That is, if we want to get all segments to be recovered before
> > running
> > > > log
> > > > > recovery, we need to break the logic in UnifiedLog, to create a
> > partial
> > > > > UnifiedLog instance, and add more info to it later, which I think is
> > > just
> > > > > making the codes more complicated.
> > > > >
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > >
> > > > >
> > > > > On Thu, May 26, 2022 at 2:57 AM Jun Rao 
> > > > wrote:
> > > > >
> > > > > > Hi, Luke,
> > > > > >
> > > > > > Thanks for the KIP. Just one comment.
> > > > > >
> > > > > > 10. For kafka.log:type=LogManager,name=remainingLogsToRecovery,
> > could
> > > > we
> > > > > > instead track the number of remaining segments? This monitors the
> > > > > progress
> > > > > > at a finer granularity and is also consistent with the thread level
> > > > > metric.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Wed, May 25, 2022 at 7:47 AM Tom Bentley 
> > > > wrote:
> > > > > >
> > > > > > > Thanks Luke! LGTM.
> > > > > > >
> > > > > > > On Sun, 22 May 2022 at 05:18, Luke Chen 
> > 

Re: [VOTE] KIP-831: Add metric for log recovery progress

[Federico Valeri]

+1 non binding

Thanks
Fede

On Thu, Jun 9, 2022 at 5:47 PM Yu Kvicii  wrote:
>
> Hi there. Thanks for the proposal
> +1 binding
>
> > On Jun 9, 2022, at 21:04, Luke Chen  wrote:
> >
> > Bump this thread.
> >
> > So far, I've got:
> > 4 +1 (non-bingin) from Ziming, Divij, James, Raman
> > 1 +1 (binding) from Tom Bentley
> >
> > Thanks for people who vote for this KIP.
> > Any other votes/comments are welcomed?
> >
> > Thank you.
> > Luke
> >
> > On Wed, May 25, 2022 at 10:55 PM Tom Bentley  wrote:
> >
> >> Hi Luke,
> >>
> >> Thanks for the KIP. +1 (binding).
> >>
> >> Kind regards,
> >>
> >> Tom
> >>
> >> On Fri, 20 May 2022 at 02:58, Raman Verma 
> >> wrote:
> >>
> >>> ^^  (non binding)
> >>>
> >>> On Thu, May 19, 2022 at 5:40 PM Raman Verma  wrote:
> 
>  Hello Luke,
> 
>  The proposal looks good to me. Thanks. +1
> 
> 
>  On Tue, May 17, 2022 at 9:26 PM James Cheng 
> >>> wrote:
> >
> > +1 (non-binding)
> >
> > -James
> >
> > Sent from my iPhone
> >
> >> On May 16, 2022, at 12:12 AM, Luke Chen  wrote:
> >>
> >> Hi all,
> >>
> >> I'd like to start a vote on KIP to expose metrics for log recovery
> >> progress. These metrics would let the admins have a way to monitor
> >>> the log
> >> recovery progress.
> >>
> >> Details can be found here:
> >>
> >>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
> >>
> >> Any feedback is appreciated.
> >>
> >> Thank you.
> >> Luke
> 
> 
> 
>  --
>  Best Regards,
>  Raman Verma
> >>>
> >>>
> >>>
> >>> --
> >>> Best Regards,
> >>> Raman Verma
> >>>
> >>>
> >>
>

Re: [VOTE] 3.2.1 RC3

[Federico Valeri]

gpg: Good signature from "David Arthur (CODE SIGNING KEY)
" [unknown]

/tmp/tmp.lRpAY/kafka-3.2.1-src.tgz.sha512: OK

Build using Gradle 7.3.3, Java 11 and Scala 2.13.8
BUILD SUCCESSFUL in 5m 34s

Unit and integration tests:
BUILD SUCCESSFUL in 37m 47s

I also used Scala 2.13 binaries and staged Maven artifacts to run some
client applications.

+1 (non binding)

Thanks


On Fri, Jul 22, 2022 at 4:21 PM Christopher Shannon
 wrote:
>
> +1 (non binding)
>
> I built from source and ran through some of the tests including all the
> connect runtime tests. I verified that KAFKA-14079 was included and the fix
> looked good in my tests.
>
> On Thu, Jul 21, 2022 at 9:15 PM David Arthur  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first release candidate of Apache Kafka 3.2.1.
> >
> > This is a bugfix release with several fixes since the release of 3.2.0. A
> > few of the major issues include:
> >
> > * KAFKA-14062 OAuth client token refresh fails with SASL extensions
> > * KAFKA-14079 Memory leak in connectors using errors.tolerance=all
> > * KAFKA-14024 Cooperative rebalance regression causing clients to get stuck
> >
> >
> > Release notes for the 3.2.1 release:
> > https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/RELEASE_NOTES.html
> >
> >
> >
> >  Please download, test and vote by Wednesday July 27, 2022 at 17:00 PT.
> > 
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/
> >
> > Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > Javadoc: https://home.apache.org/~davidarthur/kafka-3.2.1-rc3/javadoc/
> >
> > Tag to be voted upon (off 3.2 branch) is the 3.2.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.2.1-rc3
> >
> > Documentation: https://kafka.apache.org/32/documentation.html
> >
> > Protocol: https://kafka.apache.org/32/protocol.html
> >
> >
> > The past few builds have had flaky test failures. I will update this thread
> > with passing build links soon.
> >
> > Unit/Integration test job:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/
> > System test job:
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/
> >
> >
> > Thanks!
> > David Arthur
> >

Re: [VOTE] KIP-678: New Kafka Connect SMT for plainText => struct with Regex

[Federico Valeri]

+1 (non binding)

Thanks.


On Fri, Sep 2, 2022 at 5:32 PM Chris Egerton  wrote:
>
> +1 (binding)
>
> Thanks for the KIP!
>
> On Thu, Sep 1, 2022 at 9:52 PM gyejun choi  wrote:
>
> > Hi all,
> > I'd like to start a vote for KIP-678: New Kafka Connect SMT for plainText
> > => struct with Regex
> >
> > KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex
> >
> > JIRA: https://github.com/apache/kafka/pull/12219
> >
> > Discussion thread:
> > https://lists.apache.org/thread/xb57l7j953k8dfgqvktb09y31vzpm1xx
> > https://lists.apache.org/thread/7t1k0ko8l973v4oj3l983j7qpwolhyzf
> >
> > Thanks,
> > whsoul
> >

Re: [VOTE] 3.3.1 RC0

[Federico Valeri]

Hi, I did the following to validate the release:

- Checksums and signatures ok
- Build from source ok
- Unit and integration tests ok
- Quickstart in both ZK and KRaft modes ok
- Test Java app with staging Maven artifacts ok

+1 (non binding)

Thanks
Fede

On Sun, Oct 2, 2022 at 7:47 PM José Armando García Sancio
 wrote:
>
> Hi all,
>
> All of the system tests for 3.3 passed.
>
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.3/2022-09-30--001.system-test-kafka-3.3--1664605767--confluentinc--3.3--eefe867118/report.html
>
> This build ran all of the tests and there were two failures:
> kafkatest.tests.core.delegation_token_test and
> kafkatest.tests.streams.streams_broker_compatibility_test.
>
> I ran the kafkatest.tests.streams.streams_broker_compatibility_test
> module by itself and it passed:
> http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1664643010--apache--3.3--cdb25e10dc/2022-10-01--001./2022-10-01--001./report.html
>
> For kafkatest.tests.core.delegation_token_test there was an issue with
> the test that David Arthur fixed. I ran that module with that fix and
> it passed: 
> http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1664678563--apache--3.3--c2d7984c8e/2022-10-02--001./2022-10-02--001./report.html
>
> Thank you,
> --
> -José

Re: [VOTE] 3.3.2 RC1

[Federico Valeri]

Hi, I did the following to validate the release:

- Checksums and signatures ok
- Build from source using Java 17 and Scala 2.13 ok
- Unit and integration tests ok
- Quickstart in both ZK and KRaft modes ok
- Test app with staging Maven artifacts ok

+1 (non binding)

Thanks
Fede

On Wed, Dec 21, 2022 at 11:21 PM Chris Egerton  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.3.2.
>
> This is a bugfix release with several fixes since the release of 3.3.1. A
> few of the major issues include:
>
> * KAFKA-14358 Users should not be able to create a regular topic name
> __cluster_metadata
> KAFKA-14379 Consumer should refresh preferred read replica on update
> metadata
> * KAFKA-13586 Prevent exception thrown during connector update from
> crashing distributed herder
>
>
> Release notes for the 3.3.2 release:
> https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, January 6, 2023, 10pm UTC
> (this date is chosen to accommodate the various upcoming holidays that
> members of the community will be taking and give everyone enough time to
> test out the release candidate, without unduly delaying the release)
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~cegerton/kafka-3.3.2-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cegerton/kafka-3.3.2-rc1/javadoc/
>
> * Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
> https://github.com/apache/kafka/releases/tag/3.3.2-rc1
>
> * Documentation:
> https://kafka.apache.org/33/documentation.html
>
> * Protocol:
> https://kafka.apache.org/33/protocol.html
>
> The most recent build has had test failures. These all appear to be due to
> flakiness, but it would be nice if someone more familiar with the failed
> tests could confirm this. I may update this thread with passing build links
> if I can get one, or start a new release vote thread if test failures must
> be addressed beyond re-running builds until they pass.
>
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/142/testReport/
>
> José, would it be possible to re-run the system tests for 3.3 on the latest
> commit for 3.3 (e3212f2), and share the results on this thread?
>
> Cheers,
>
> Chris

Re: [ANNOUNCE] New committer: Satish Duggana

[Federico Valeri]

Hi Satish, congrats!

On Fri, Dec 23, 2022, 8:46 PM Viktor Somogyi-Vass
 wrote:

> Congrats Satish!
>
> On Fri, Dec 23, 2022, 19:38 Mickael Maison 
> wrote:
>
> > Congratulations Satish!
> >
> > On Fri, Dec 23, 2022 at 7:36 PM Divij Vaidya 
> > wrote:
> > >
> > > Congratulations Satish! 
> > >
> > > On Fri 23. Dec 2022 at 19:32, Josep Prat 
> > > wrote:
> > >
> > > > Congrats Satish!
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > <
> >
> https://www.google.com/maps/search/Immanuelkirchstra%C3%9Fe+26,+10405+Berlin?entry=gmail=g
> > >
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Fri, Dec 23, 2022, 19:23 Chris Egerton 
> > wrote:
> > > >
> > > > > Congrats, Satish!
> > > > >
> > > > > On Fri, Dec 23, 2022, 13:19 Arun Raju 
> wrote:
> > > > >
> > > > > > Congratulations 
> > > > > >
> > > > > > On Fri, Dec 23, 2022, 1:08 PM Jun Rao 
> > > > wrote:
> > > > > >
> > > > > > > Hi, Everyone,
> > > > > > >
> > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > committer
> > > > > > Satish
> > > > > > > Duggana.
> > > > > > >
> > > > > > > Satish has been a long time Kafka contributor since 2017. He is
> > the
> > > > > main
> > > > > > > driver behind KIP-405 that integrates Kafka with remote
> storage,
> > a
> > > > > > > significant and much anticipated feature in Kafka.
> > > > > > >
> > > > > > > Congratulations, Satish!
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > --
> > > Divij Vaidya
> >
>

Re: [VOTE] 3.3.2 RC0

[Federico Valeri]

Hi, I did the following to validate the release:

- Checksums and signatures ok
- Build from source using Java 17 and Scala 2.13 ok
- Unit and integration tests ok
- Quickstart in both ZK and KRaft modes ok
- Test app with staging Maven artifacts ok

Documentation still has 3.3.1 version references, but I guess this
will be updated later.

+1 (non binding)

Thanks
Fede


On Fri, Dec 16, 2022 at 11:51 AM jacob bogers  wrote:
>
> Hi, I have tried several times to unsub from dev@kafka.apache.org, but it
> just isnt working
>
> Can someone help me?
>
> cheers
> Jacob
>
> On Thu, Dec 15, 2022 at 5:37 PM Chris Egerton 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.3.2.
> >
> > This is a bugfix release with several fixes since the release of 3.3.1. A
> > few of the major issues include:
> >
> > * KAFKA-14358 Users should not be able to create a regular topic name
> > __cluster_metadata
> > KAFKA-14379 Consumer should refresh preferred read replica on update
> > metadata
> > * KAFKA-13586 Prevent exception thrown during connector update from
> > crashing distributed herder
> >
> >
> > Release notes for the 3.3.2 release:
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc0/RELEASE_NOTES.html
> >
> >
> >
> > *** Please download, test and vote by Tuesday, December 20, 10pm UTC
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc0/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cegerton/kafka-3.3.2-rc0/javadoc/
> >
> > * Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
> > https://github.com/apache/kafka/releases/tag/3.3.2-rc0
> >
> > * Documentation:
> > https://kafka.apache.org/33/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/33/protocol.html
> >
> > The most recent build has had test failures. These all appear to be due to
> > flakiness, but it would be nice if someone more familiar with the failed
> > tests could confirm this. I may update this thread with passing build links
> > if I can get one, or start a new release vote thread if test failures must
> > be addressed beyond re-running builds until they pass.
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/135/testReport/
> >
> > System tests:
> >
> > http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1670984851--apache--3.3--22af3f29ce/2022-12-13--001./2022-12-13--001./report.html
> > (initial with three flaky failures)
> > Follow-up system tests:
> > https://home.apache.org/~cegerton/system_tests/2022-12-14--015/report.html
> > ,
> > https://home.apache.org/~cegerton/system_tests/2022-12-14--016/report.html
> > ,
> >
> > http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1671061000--apache--3.3--69fbaf2457/2022-12-14--001./2022-12-14--001./report.html
> >
> > (Note that the exact commit used for some of the system test runs will not
> > precisely match the commit for the release candidate, but that all
> > differences between those two commits should have no effect on the
> > relevance or accuracy of the test results.)
> >
> > Thanks,
> >
> > Chris
> >

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

[Federico Valeri]

Hi Michael, thanks for the KIP. Looks great.

I noticed that there is a difference on how MetricName and Sensor are
created and removed in PluginMetrics class:

- In metricName() it is not clear what happens if the MetricName
already exists, I guess it will be the same "get or create" behavior
we have on sensor(), but maybe we could clarify in the comment.
- In removeSensor() we do not return any value, so one can't tell if
the Sensor was there or not like with removeMetric().

Minor issue: in the "Example usage" we have setPluginMetrics override,
but should be withPluginMetrics according to the Monitorable
interface.

On Mon, Jan 23, 2023 at 9:01 PM Chris Egerton  wrote:
>
> Hi Mickael,
>
> Thanks for the updates, this is looking great.
>
> I have two more small thoughts:
>
> 1) What's the rationale for defining PluginMetrics as a class instead of an
> interface? AFAICT we don't need a public constructor for it since the
> runtime will be responsible for creating all instances.
>
> 2) The list of affected plugin classes is indeed quite long--thanks for
> listing them all out! I noticed that the ReplicationPolicy interface isn't
> listed for MM2. Is this intentional?
>
> Cheers,
>
> Chris
>
> On Fri, Jan 13, 2023 at 12:26 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I've updated the KIP based on the feedback.
> >
> > Now instead of receiving a Metrics instance, plugins get access to
> > PluginMetrics that exposes a much smaller API. I've removed the
> > special handling for connectors and tasks and they must now implement
> > the Monitorable interface as well to use this feature. Finally the
> > goal is to allow all plugins (apart from metrics reporters) to use
> > this feature. I've listed them all (there are over 30 pluggable APIs)
> > but I've not added the list in the KIP. The reason is that new plugins
> > could be added in the future and instead I'll focus on adding support
> > in all the place that instantiate classes.
> >
> > Thanks,
> > Mickael
> >
> > On Tue, Jan 10, 2023 at 7:00 PM Mickael Maison 
> > wrote:
> > >
> > > Hi Chris/Yash,
> > >
> > > Thanks for taking a look and providing feedback.
> > >
> > > 1) Yes you're right, when using incompatible version, metrics() would
> > > trigger NoSuchMethodError. I thought using the context to pass the
> > > Metrics object would be more idiomatic for Connect but maybe
> > > implementing Monitorable would be simpler. It would also allow other
> > > Connect plugins (transformations, converters, etc) to register
> > > metrics. So I'll make that change.
> > >
> > > 2) As mentioned in the rejected alternatives, I considered having a
> > > PluginMetrics class/interface with a limited API. But since Metrics is
> > > part of the public API, I thought it would be simpler to reuse it.
> > > That said you bring interesting points so I took another look today.
> > > It's true that the Metrics API is pretty complex and most methods are
> > > useless for plugin authors. I'd expect most use cases only need one
> > > addMetric and one sensor methods. Rather than subclassing Metrics, I
> > > think a delegate/forwarding pattern might work well here. A
> > > PluginMetric class would forward its method to the Metrics instance
> > > and could perform some basic validations such as only letting plugins
> > > delete metrics they created, or automatically injecting tags with the
> > > class name for example.
> > >
> > > 3) Between the clients, brokers, streams and connect, Kafka has quite
> > > a lot! In practice I think registering metrics should be beneficial
> > > for all plugins, I think the only exception would be metrics reporters
> > > (which are instantiated before the Metrics object). I'll try to build
> > > a list of all plugin types and add that to the KIP.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Tue, Dec 27, 2022 at 4:54 PM Chris Egerton 
> > wrote:
> > > >
> > > > Hi Yash,
> > > >
> > > > Yes, a default no-op is exactly what I had in mind should the
> > Connector and
> > > > Task classes implement the Monitorable interface.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Dec 20, 2022 at 2:46 AM Yash Mayya 
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for creating this KIP, this will be a super useful feature to
> > > > > enhance existing connectors in the Kafka Connect ecosystem.
> > > > >
> > > > > I have some similar concerns to the ones that Chris has outlined
> > above,
> > > > > especially with regard to directly exposing Connect's Metrics object
> > to
> > > > > plugins. I believe it would be a lot friendlier to developers if we
> > instead
> > > > > exposed wrapper methods in the context classes - such as one for
> > > > > registering a new metric, one for recording metric values and so on.
> > This
> > > > > would also have the added benefit of minimizing the surface area for
> > > > > potential misuse by custom plugins.
> > > > >
> > > > > > for connectors and tasks they should handle 

Re: [DISCUSS] KIP-898: Modernize Connect plugin discovery

[Federico Valeri]

ut the following points don't appear to be related.
> >
> > 3. (Nit) Could we rename the migration script something like
> > "connect-migrate-plugin-path" or even just "connect-plugin-path" with a
> > "migrate" subcommand?
> >
> > 4. It's noted that "If a plugin class is removed from the path, the
> > corresponding shim manifest should also be removed". Did you consider
> > adding behavior (possibly guarded behind a CLI flag) to find and remove
> > manifests for nonexistent plugins?
> >
> > 5. What's the concrete plan for updating the plugins that ship OOTB with
> > Connect? Will they be given a service loader manifest, a module info file,
> > or both?
> >
> > 6. In the HYBRID_WARN mode, what precedence will we give to plugins that
> > are found by both the legacy scanning logic and the new service
> > loader/module info loading logic?
> >
> > 7. It's a bit strange to immediately deprecate config property values and
> > the migration script in their first release. IMO as long as we have the
> > migration script in the project, it doesn't make sense to deprecate it, and
> > until we've had at least one successful release with the new loading logic
> > and some confidence in it, we should not deprecate the older logic. I think
> > a more reasonable pace for the update here could be to deprecate the legacy
> > scanning logic in 4.0 (changing the default accordingly) and remove it
> > entirely in 5.0.
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Jan 23, 2023 at 3:21 AM Federico Valeri 
> > wrote:
> >
> > > Hi Greg, this looks like a useful change to me.
> > >
> > > I was just wondering if we should use a better script name like
> > > "connect-convert-to-service-provider.sh" or something like this, and
> > > maybe add a --dry-run option.
> > >
> > > On Tue, Jan 17, 2023 at 8:45 PM Greg Harris
> > >  wrote:
> > > >
> > > > Hi all!
> > > >
> > > > I'd like to start a discussion about
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-898%3A+Modernize+Connect+plugin+discovery
> > > > .
> > > >
> > > > Thanks!
> > > > Greg Harris
> > >
> >

Re: [DISCUSS] KIP-898: Modernize Connect plugin discovery

[Federico Valeri]

Hi Greg, this looks like a useful change to me.

I was just wondering if we should use a better script name like
"connect-convert-to-service-provider.sh" or something like this, and
maybe add a --dry-run option.

On Tue, Jan 17, 2023 at 8:45 PM Greg Harris
 wrote:
>
> Hi all!
>
> I'd like to start a discussion about
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-898%3A+Modernize+Connect+plugin+discovery
> .
>
> Thanks!
> Greg Harris

Re: [VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Federico Valeri]

+1 (non binding)

Thanks.

On Thu, Jan 26, 2023 at 2:30 PM Gantigmaa Selenge  wrote:
>
> Hi
>
> I'd like to call for a vote on KIP-894, which updates MirrorMaker to use
> IncrementalAlterConfigs API to sync topic configurations between clusters.
>
> The KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
>
> The discussion thread:
> https://lists.apache.org/thread/4chr4s5vbd9rqhml2tk60fsghojwo6bb
>
> Thank you
> Tina

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Federico Valeri]

On Wed, Jan 11, 2023 at 12:03 PM Gantigmaa Selenge  wrote:
>
> Thank you both for the feedback.
>
> > Have we considered using incremental alter configs by
> default and fallback to the legacy one if the former is unavailable?
> Initially having a flag with just on and off switches seemed simpler and it
> gives users control and awareness of what's being used. However, now I
> think using incremental alter configs by default and if it is unavailable,
> fallback to the legacy API is a good idea.
>
> > The config could have 3 possible values: requested, required, never. The
> default would be requested.
>
> I do like this idea, however when it's set to required, I wasn't sure how
> the mirrormaker should have. It's probably not a great experience if
> mirrormaker starts crashing at some point after it's already running due to
> an incompatible broker version.
>
> If the incrementalAlterConfig request fails because the target cluster is
> running an older version, then we could log a WARN message that says
> something like  "The config to use incrementalAlterConfig API for syncing
> topic configurations has been set to true however target cluster is running
> incompatible version therefore using the legacy alterConfig API". This way
> the Mirrormaker never has to stop working and makes the user aware of
> what's being used.  In this case, we also would not need 3 separate values
> for the config, instead would use the original true or false values:
> - true - > would use incrementalAlterConfig API, but if it's unavailable,
> fallback to the legacy API
> - false -> keep using the legacy API
>
> Set the flag to true by default and remove the config in 4.0.

Hi Tina, I'm in favor of this variant. Maybe the warning message could
just be "The target cluster  is not compatible with the
IncrementalAlterConfigs API, falling back to AlterConfigs API".

Thanks.

>
> > One suggestion: I'm not sure how concerned you are about people's ability
> to migrate, but if you want to make it as smooth as possible, you could add
> one more step. In the 4.0 release, while removing
> `use.incremental.alter.configs`, you can also add
> `use.legacy.alter.configs` to give users a path to continue using the old
> behavior even after the default changes.
>
> If we implement the fallback logic as Ismael suggested above, I think we
> would not need this extra flag later anymore.
>
> Please let me know what you think. Then I can go ahead and update the KIP.
>
> Regards,
> Tina
>
> On Sat, Jan 7, 2023 at 7:45 PM Ismael Juma  wrote:
>
> > Hi,
> >
> > Thanks for the KIP. Have we considered using incremental alter configs by
> > default and fallback to the legacy one if the former is unavailable?
> >
> > The config could have 3 possible values: requested, required, never. The
> > default would be requested. In 4.0, this could would be removed and it
> > would effectively be required.
> >
> > Ismael
> >
> > On Sat, Jan 7, 2023, 10:03 AM John Roesler  wrote:
> >
> > > Hi Tina,
> > >
> > > Thanks for the KIP!
> > >
> > > I hope someone with prior MM or Kafka config experience is able to chime
> > > in here; I have neither.
> > >
> > > I took a look at your KIP, and it makes sense to me. I also think your
> > > migration plan is a good one.
> > >
> > > One suggestion: I'm not sure how concerned you are about people's ability
> > > to migrate, but if you want to make it as smooth as possible, you could
> > add
> > > one more step. In the 4.0 release, while removing
> > > `use.incremental.alter.configs`, you can also add
> > > `use.legacy.alter.configs` to give users a path to continue using the old
> > > behavior even after the default changes.
> > >
> > > Clearly, this will prolong the deprecation period, with implications on
> > > code maintenance, so there is some downside. But generally, I've found
> > > going above and beyond to support smooth upgrades for users to be well
> > > worth it in the long run.
> > >
> > > Thanks again,
> > > -John
> > >
> > >
> > > On Fri, Jan 6, 2023, at 05:49, Gantigmaa Selenge wrote:
> > > > Hi everyone,
> > > >
> > > > I would like to start a discussion on the MirrorMaker update that
> > > > proposes
> > > > replacing the deprecated alterConfigs API with the
> > > > incrementalAlterConfigs
> > > > API for syncing topic configurations. Please take a look at the
> > proposal
> > > > here:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
> > > >
> > > >
> > > > Regards,
> > > > Tina
> > >
> >

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Federico Valeri]

+ 1 (non binding) for the latest revision of this KIP (RecordReader).

Thanks
Fede

On Sat, Feb 25, 2023 at 7:04 AM Luke Chen  wrote:
>
> +1 from me.
>
> Thank you
> Luke
>
> On Sat, Feb 25, 2023 at 8:39 AM Chia-Ping Tsai  wrote:
>
> > Dear all,
> >
> > All comments are addressed , and so please take look at this KIP. It needs
> > vote and love :)
> >
> > thanks.

Re: [VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Bumping this thread for more votes.

Thanks.

On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez 
wrote:

> Hi, Frederico,
>
> Thanks for the KIP.
>
> Non-binding +1.
>
> Thanks,
> Alexandre
>
> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> >
> > +1 from me.
> >
> > Luke
> >
> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > >
> > > Thanks
> > > Fede
> > >
>

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Bumping this thread as KIP-641 is now adopted.

Thanks
Fede

On Thu, Feb 16, 2023 at 11:13 AM Federico Valeri  wrote:
>
> On Thu, Feb 16, 2023 at 10:26 AM Luke Chen  wrote:
> >
> > Hi Federico,
> >
> > 1. Sounds good.
> > I think you should make it clear in the KIP.
> >
> > We only have one tool with this issue (kafka.tools.StateChangeLogMerger),
> > which is tracked in KAFKA-7735
> > <https://issues.apache.org/jira/browse/KAFKA-7735>. This has been broken
> > since 2.0 release, so it means that there is not much interest and we
> > should simply deprecate.
> > It makes me think we're going to abandon this tool since no one is
> > interested in it.
> > Could we make it clear to mention the broken tool will be fixed in an
> > on-going PR, but not included in the scope of this KIP?
> >
>
> Sure. That's a good idea.
>
> > 2. For the SPI argument, currently, KIP-641 is still under discussion, not
> > passing the votes.
> > Do we have any alternative if KIP-641 is rejected?
> >
>
> I think the first rejected alternative could also be used for the SPI
> argument issue.
>
> > Thank you.
> > Luke
> >
> > On Thu, Feb 16, 2023 at 4:56 PM Federico Valeri 
> > wrote:
> >
> > > Hi Luke, thanks for the feedback.
> > >
> > > On Thu, Feb 16, 2023 at 9:33 AM Luke Chen  wrote:
> > > >
> > > > Hi Federico,
> > > >
> > > > Thanks for the KIP.
> > > > It's really helpful to have a general guideline for tools migration so
> > > that
> > > > contributors and reviewers can be clear what to be expected.
> > > >
> > > > One question:
> > > > In the broker link tool, you said we should directly deprecate it.
> > > > But does that mean this tool will keep broken?
> > >
> > > No, as I already did both migration and fix in the following PR:
> > >
> > > https://github.com/apache/kafka/pull/13171
> > >
> > > > Will we fix that during the module movement?
> > > >
> > >
> > > If the above PR is accepted, we will have a deprecated but working
> > > version. The general orientation is to remove it in the next major
> > > release, because it was broker for a long time and nobody complained.
> > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Thu, Feb 16, 2023 at 4:16 PM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Sorry for the spam. I had some issue with my mail client.
> > > > >
> > > > > On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri 
> > > > > wrote:
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I opened KIP-906 to have some guidelines around tools migration:
> > > > > >
> > > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > > > > >
> > > > > > Let me know if you have any feedback or suggestions.
> > > > > >
> > > > > > Thanks
> > > > > > Fede
> > > > >
> > >

[VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I'd like to start the vote on KIP-906 Tools migration guidelines.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Discussion thread:
https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p

Thanks
Fede

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Federico Valeri]

Hi Chia-Ping, thanks for updating the KIP.

I would only add that we plan to remove the old trait in the next
major release. I think it's better to be explicit about this.

On Thu, Feb 16, 2023 at 11:34 AM Chia-Ping Tsai  wrote:
>
> Dear all,
>
> I have updated the KIP to address comments. PTAL
>
>  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

On Thu, Feb 16, 2023 at 10:26 AM Luke Chen  wrote:
>
> Hi Federico,
>
> 1. Sounds good.
> I think you should make it clear in the KIP.
>
> We only have one tool with this issue (kafka.tools.StateChangeLogMerger),
> which is tracked in KAFKA-7735
> <https://issues.apache.org/jira/browse/KAFKA-7735>. This has been broken
> since 2.0 release, so it means that there is not much interest and we
> should simply deprecate.
> It makes me think we're going to abandon this tool since no one is
> interested in it.
> Could we make it clear to mention the broken tool will be fixed in an
> on-going PR, but not included in the scope of this KIP?
>

Sure. That's a good idea.

> 2. For the SPI argument, currently, KIP-641 is still under discussion, not
> passing the votes.
> Do we have any alternative if KIP-641 is rejected?
>

I think the first rejected alternative could also be used for the SPI
argument issue.

> Thank you.
> Luke
>
> On Thu, Feb 16, 2023 at 4:56 PM Federico Valeri 
> wrote:
>
> > Hi Luke, thanks for the feedback.
> >
> > On Thu, Feb 16, 2023 at 9:33 AM Luke Chen  wrote:
> > >
> > > Hi Federico,
> > >
> > > Thanks for the KIP.
> > > It's really helpful to have a general guideline for tools migration so
> > that
> > > contributors and reviewers can be clear what to be expected.
> > >
> > > One question:
> > > In the broker link tool, you said we should directly deprecate it.
> > > But does that mean this tool will keep broken?
> >
> > No, as I already did both migration and fix in the following PR:
> >
> > https://github.com/apache/kafka/pull/13171
> >
> > > Will we fix that during the module movement?
> > >
> >
> > If the above PR is accepted, we will have a deprecated but working
> > version. The general orientation is to remove it in the next major
> > release, because it was broker for a long time and nobody complained.
> >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Feb 16, 2023 at 4:16 PM Federico Valeri 
> > > wrote:
> > >
> > > > Sorry for the spam. I had some issue with my mail client.
> > > >
> > > > On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri 
> > > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I opened KIP-906 to have some guidelines around tools migration:
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > > > >
> > > > > Let me know if you have any feedback or suggestions.
> > > > >
> > > > > Thanks
> > > > > Fede
> > > >
> >

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Sorry for the spam. I had some issue with my mail client.

On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri  wrote:
>
> Hi everyone,
>
> I opened KIP-906 to have some guidelines around tools migration:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
>
> Let me know if you have any feedback or suggestions.
>
> Thanks
> Fede

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi Luke, thanks for the feedback.

On Thu, Feb 16, 2023 at 9:33 AM Luke Chen  wrote:
>
> Hi Federico,
>
> Thanks for the KIP.
> It's really helpful to have a general guideline for tools migration so that
> contributors and reviewers can be clear what to be expected.
>
> One question:
> In the broker link tool, you said we should directly deprecate it.
> But does that mean this tool will keep broken?

No, as I already did both migration and fix in the following PR:

https://github.com/apache/kafka/pull/13171

> Will we fix that during the module movement?
>

If the above PR is accepted, we will have a deprecated but working
version. The general orientation is to remove it in the next major
release, because it was broker for a long time and nobody complained.

> Thank you.
> Luke
>
> On Thu, Feb 16, 2023 at 4:16 PM Federico Valeri 
> wrote:
>
> > Sorry for the spam. I had some issue with my mail client.
> >
> > On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri 
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > I opened KIP-906 to have some guidelines around tools migration:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > >
> > > Let me know if you have any feedback or suggestions.
> > >
> > > Thanks
> > > Fede
> >

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Federico Valeri]

+ 1 (non binding)

Thanks

On Sat, Feb 18, 2023 at 9:36 AM Chia-Ping Tsai  wrote:
>
> Hi,
>
> I'd like to start the vote on KIP-614: An new java interface to replace 
> 'kafka.common.MessageReader'
>
> KIP-614: 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866569
>
> thread: 
> https://lists.apache.org/thread.html/r6db6708f64345bb8fe0d573e05014fb790e69d501f21f855ca65619a%40%3Cdev.kafka.apache.org%3E
>
> Cheers,
> Chia-Ping

Re: [VOTE] KIP-898: Modernize Connect plugin discovery

[Federico Valeri]

+1 (non binding)

Thanks
Fede

On Tue, Feb 28, 2023 at 10:10 AM Mickael Maison
 wrote:
>
> +1 (binding)
>
> Thanks,
> Mickael
>
> On Mon, Feb 27, 2023 at 7:42 PM Chris Egerton  wrote:
> >
> > +1 (binding). Thanks for the KIP!
> >
> > On Mon, Feb 27, 2023 at 12:51 PM Greg Harris 
> > wrote:
> >
> > > Hi,
> > >
> > > I'd like to call a vote for KIP-898 which aims to improve the performance
> > > of Connect startup by allowing discovery of plugins via the ServiceLoader.
> > >
> > > KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-898
> > > %3A+Modernize+Connect+plugin+discovery
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/wxh0r343w86s91py0876njzbyn5qxd8s
> > >
> > > Thanks!
> > >

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Federico Valeri]

Congrats Mickael, very well deserved, and thanks Jun for covering this
role so far.

On Fri, Apr 21, 2023 at 5:09 PM Jun Rao  wrote:
>
> Hi, everyone,
>
> After more than 10 years, I am stepping down as the PMC chair of Apache
> Kafka. We now have a new chair Mickael Maison, who has been a PMC member
> since 2020. I plan to continue to contribute to Apache Kafka myself.
>
> Congratulations, Mickael!
>
> Jun

Re: [VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
time to review the KIP. Can I get one more binding vote?

Cheers
Fede

On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
>
> Thanks Federico,
>
> I'm +1 (binding)
>
> -John
>
> On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > Bumping this thread for more votes.
> >
> > Thanks.
> >
> > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez 
> > 
> > wrote:
> >
> >> Hi, Frederico,
> >>
> >> Thanks for the KIP.
> >>
> >> Non-binding +1.
> >>
> >> Thanks,
> >> Alexandre
> >>
> >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> >> >
> >> > +1 from me.
> >> >
> >> > Luke
> >> >
> >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
> >> > wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> >> > >
> >> > >
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> >> > >
> >> > > Discussion thread:
> >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> >> > >
> >> > > Thanks
> >> > > Fede
> >> > >
> >>

Re: [VOTE] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi, the KIP is accepted with 3 binding votes (Luke, John and Tom) and
1 non binding vote (Alexandre).

Thanks everyone!

Cheers
Fede

On Tue, Apr 4, 2023 at 1:58 PM Tom Bentley  wrote:
>
> Hi Federico,
>
> Thanks for the KIP, +1 (binding).
>
> Tom
>
> On Tue, 4 Apr 2023 at 08:30, Federico Valeri  wrote:
>
> > Hi, we have 2 binding an 1 non binding votes. Thanks for tacking the
> > time to review the KIP. Can I get one more binding vote?
> >
> > Cheers
> > Fede
> >
> > On Fri, Mar 24, 2023 at 3:05 PM John Roesler  wrote:
> > >
> > > Thanks Federico,
> > >
> > > I'm +1 (binding)
> > >
> > > -John
> > >
> > > On Fri, Mar 24, 2023, at 01:11, Federico Valeri wrote:
> > > > Bumping this thread for more votes.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, Mar 15, 2023, 9:57 AM Alexandre Dupriez <
> > alexandre.dupr...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi, Frederico,
> > > >>
> > > >> Thanks for the KIP.
> > > >>
> > > >> Non-binding +1.
> > > >>
> > > >> Thanks,
> > > >> Alexandre
> > > >>
> > > >> Le mer. 15 mars 2023 à 08:28, Luke Chen  a écrit :
> > > >> >
> > > >> > +1 from me.
> > > >> >
> > > >> > Luke
> > > >> >
> > > >> > On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri <
> > fedeval...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > I'd like to start the vote on KIP-906 Tools migration guidelines.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > > >> > >
> > > >> > > Discussion thread:
> > > >> > > https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
> > > >> > >
> > > >> > > Thanks
> > > >> > > Fede
> > > >> > >
> > > >>
> >
> >

Re: [VOTE] 3.4.0 RC2

[Federico Valeri]

+1 (non binding)

- Ran the unit and integration test suites with Java 17 and Scala 2.13
- Ran a series of basic examples and client configurations
- Spot checked the docs and Javadocs

Thanks
Fede

On Fri, Feb 3, 2023 at 5:29 PM Jakub Scholz  wrote:
>
> +1 (non-binding). I run my tests with the staged Scala 2.13 binaries and
> staged Maven artifacts. All seems to work fine.
>
> Thanks & Regards
> Jakub
>
> On Tue, Jan 31, 2023 at 8:01 PM David Arthur  wrote:
>
> > Hey folks, we found a couple of blockers with RC1 and have fixed them in
> > the latest release candidate, RC2.
> >
> > The major features of this release include:
> >
> > * KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers
> > >
> >
> > * KIP-876: Time based cluster metadata snapshots
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-876%3A+Time+based+cluster+metadata+snapshots
> > >
> >
> > * KIP-787: MM2 manage Kafka resources with custom Admin implementation.
> > <
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191335620
> > >
> >
> > * KIP-866 ZooKeeper to KRaft Migration
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration
> > >
> > (Early
> > Access)
> >
> >
> >
> > Release notes for the 3.4.0 release:
> >
> > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/RELEASE_NOTES.html
> >
> >
> > Please download, test and vote by Friday, February 3, 5pm PT
> >
> >
> > ---
> >
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> >
> > https://kafka.apache.org/KEYS
> >
> >
> > * Release artifacts to be voted upon (source and binary):
> >
> > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/
> >
> >
> > * Maven artifacts to be voted upon:
> >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> >
> > * Javadoc:
> >
> > https://home.apache.org/~davidarthur/kafka-3.4.0-rc2/javadoc/
> >
> >
> > * Tag to be voted upon (off 3.4 branch) is the 3.4.0 tag:
> >
> > https://github.com/apache/kafka/releases/tag/3.4.0-rc2
> >
> >
> > * Documentation:
> >
> > https://kafka.apache.org/34/documentation.html
> >
> >
> > * Protocol:
> >
> > https://kafka.apache.org/34/protocol.html
> >
> >
> > ---
> >
> >
> > Test results:
> >
> >
> > We haven't had a 100% passing build, but the latest system test run looks
> > pretty good:
> >
> > http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.4/2023-01-31--001.system-test-kafka-3.4--1675184554--confluentinc--3.4--ef3f5bd834/report.html
> >
> >
> > Here are the Jenkins test runs for 3.4:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/. We will
> > continue
> > trying to diagnose the flaky test failures as the release continues. I do
> > not expect that any of these test failures are blockers for the release.
> >
> >
> > Thanks!
> >
> > David Arthur
> >

Re: [DISCUSS] KIP-888: Batch describe ACLs and describe client quotas

[Federico Valeri]

Hi Michael, thanks for the KIP.

In addition to the deduplication logic, I wonder if we should also
consider introducing a pagination mechanism. Talking about ACLs, there
could be many rules for each user and we may have lots of users on big
clusters.



On Tue, Jan 31, 2023 at 3:58 PM Mickael Maison  wrote:
>
> Hi Tom,
>
> Thanks for the feedback, you raised a few interesting points.
>
> Regarding the response size issue: With some APIs, if the same filter
> is in the request multiple times, the response will only include it
> once. To do so you need to include the filter in the response. For
> example, this works with describeConsumerGroups because the filter,
> the group id, is small. For ACLs and Quotas, I thought having the
> exact same filters multiple times in the requests should be rare so I
> privileged returning one entry for each filter in the request, and
> avoid including the filters back in the response.
>
> One option to limit potential abuse is to restrict the accepted
> filters. For example brokers could reject requests containing
> duplicate filters. Going further we could restrict the types of
> filters allowed and only allow requests with multiple filters to have
> multiple exact match filters, or if using fuzzy matches only allow it
> on different types. I don't think this would limit the usability of
> these APIs too much.
>
> Regarding using ids for each ACL: Yes StandardAuthorizer identifies
> unique ACLs with Uuids but custom Authorizer implementations don't
> necessarily do that. So while this could help reduce response size
> this would require significant changes to the Authorizer interface.
>
> Thanks,
> Mickael
>
>
>
>
> On Fri, Dec 2, 2022 at 6:55 PM Tom Bentley  wrote:
> >
> > Hi Mickael,
> >
> > Thanks for the KIP. I can understand the motivation, but to be honest I
> > have a doubt about this.
> >
> > Taking the ACLs first, by allowing multiple filters with the current
> > proposal isn't there the chance that the same ACL will match multiple
> > filters, and be returned multiple times in the response. In fact, in the
> > worst case couldn't the client (by intent or accident) just request all
> > ACLs be included in the response an arbitrary number of times? This could
> > result in some pretty large responses. One way to avoid inflating the
> > response like this would be for the broker to deduplicate the ACLs in the
> > response by assigning an id for each, serialising each once and then using
> > the id to enumerate the matches for each pattern. It's worth noting that
> > the AccessControlEntryRecord used for KRaft clusters already has a Uuid. So
> > for the KRaft case it might be possible to use this, rather than the broker
> > having to deduplicate when handing the request.
> >
> > Another wrinkle is that if the broker calls
> > Authorizer.acls(AclBindingFilter) once for each pattern there's no
> > guarantee that the results are consistent (they could be modified between
> > calls). It could be made consistent with appropriate locking, but since in
> > practice this would be a very rare occurrence maybe we could just document
> > that as a limitation.
> >
> > Thanks again,
> >
> > Tom
> >
> > On Fri, 18 Nov 2022 at 18:00, Mickael Maison 
> > wrote:
> >
> > > Hi,
> > >
> > > I have opened KIP-888 to allow describing ACLs and Quotas in batches:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-888%3A+Batch+describe+ACLs+and+describe+client+quotas
> > >
> > > Let me know if you have any feedback or suggestions.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

[DISCUSS] KIP-906 Tools migration guidelines

[Federico Valeri]

Hi everyone,

I opened KIP-906 to have some guidelines around tools migration:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines

Let me know if you have any feedback or suggestions.

Thanks
Fede

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

Hi, one way to avoid the "fix the bug property" would be to provide
and document an additional LegacyReplicationPolicy, but still keeping
the current DefaultReplicationPolicy as replication.policy.class
default value, which is basically one of the workarounds suggested in
the KIP.

On Tue, Jul 18, 2023 at 9:49 PM Chris Egerton  wrote:
>
> Hi Federico/Omnia,
>
> Generally I like the idea of deprecating and eventually removing "fix the
> bug" properties like this, but 4.0 may be a bit soon. I'm also unsure of
> how we would instruct users who are relying on the property being set to
> "false" to migrate to a version of MM2 that doesn't support support it,
> beyond simply implementing their own replication policy--at which point,
> would we really be doing anyone a favor by forcing them to fork the default
> policy just to preserve a property we removed?
>
> I guess right now I'd rather play it safe and not immediately deprecate the
> property. If we can find an easy migration path for users who are relying
> on it, then I'd be happy to deprecate and schedule for removal.
>
> Cheers,
>
> Chris
>
> On Tue, Jul 18, 2023 at 12:54 PM Omnia Ibrahim 
> wrote:
>
> > Hi Federico,
> > I don't have any strong opinion one way or the other. The pro of
> > deprecation is not adding more configs to MM2 as it has too many configs
> > already. However, we need to think about old MM2 migrating their internal
> > topics to 4.0 with less impact.
> >
> > @Chris what do you think?
> >
> > Cheers
> > Omnia
> >
> > On Fri, Jul 14, 2023 at 2:38 PM Federico Valeri 
> > wrote:
> >
> > > Hi Omnia and Chris, I agree with setting
> > > "replication.policy.internal.topic.separator.enabled=true" by default,
> > > but I was wondering if we should also deprecate and remove in Kafka 4.
> > > If there is agreement in having the same behavior for internal and
> > > non-internal topics, then it should be fine, and we won't need to keep
> > > an additional configuration around. Wdyt?
> > >
> > > Cheers
> > > Fede
> > >
> > > On Fri, Jul 14, 2023 at 1:51 PM Omnia Ibrahim 
> > > wrote:
> > > >
> > > > Hi Chris, I added a section for backport plan here
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy#KIP949:AddflagtoenabletheusageoftopicseparatorinMM2DefaultReplicationPolicy-Backportingplan
> > > >
> > > > Cheers,
> > > > Omnia
> > > >
> > > > > On 13 Jul 2023, at 19:33, Chris Egerton 
> > > wrote:
> > > > >
> > > > > Hi Omnia,
> > > > >
> > > > > Yes, I think we ought to state the backport plan in the KIP, since
> > it's
> > > > > highly unusual for KIP changes or new configuration properties to be
> > > > > backported and we should get the approval of the community (including
> > > > > binding and non-binding voters) before implementing it.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Thu, Jul 13, 2023 at 7:13 AM Omnia Ibrahim <
> > o.g.h.ibra...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Chris,
> > > > >> The implementation should be very small so backporting this to 3.1
> > > and 3.2
> > > > >> would be perfect for this case if you or any other committer are
> > okay
> > > with
> > > > >> approving the backporting. Do we need to state this in KIP as well
> > or
> > > not?
> > > > >>
> > > > >> Also, I’ll open a vote for the KIP today and prepare the pr for it
> > so
> > > we
> > > > >> can merge it as soon as we can.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Omnia
> > > > >>
> > > > >> On Wed, Jul 12, 2023 at 4:31 PM Chris Egerton
> >  > > >
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Omnia,
> > > > >>>
> > > > >>> Thanks for changing the default, LGTM 
> > > > >>>
> > > > >>> As far as backporting goes, we probably won't be doing another
> > > release
> > > > >> 

Re: [ANNOUNCE] New committer: Greg Harris

[Federico Valeri]

Congrats Greg!

On Tue, Jul 11, 2023 at 3:55 AM Luke Chen  wrote:
>
> Congrats Greg!
>
> Luke
>
> On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
>  wrote:
>
> > Congratulations, well deserved!
> >
> > On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton  wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> > and
> > > we are happy to announce that he has accepted!
> > >
> > > Greg has been contributing to Kafka since 2019. He has made over 50
> > commits
> > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > contributions include KIP-898: "Modernize Connect plugin discovery" and a
> > > deep overhaul of the offset syncing logic in MM2 that addressed several
> > > technically-difficult, long-standing, high-impact issues.
> > >
> > > He has also been an active participant in discussions and reviews on the
> > > mailing lists and on GitHub.
> > >
> > > Thanks for all of your contributions, Greg. Congratulations!
> > >
> >
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> >

Re: [REVIEW REQUEST] KAFKA-14591 DeleteRecordsCommand moved to tools

[Federico Valeri]

HI Nikolay, I did the review.

Thanks again for working on this.

Br
Fede

On Mon, Jul 10, 2023 at 12:08 PM Николай Ижиков  wrote:
>
> Hello.
>
> I prepared PR [1] for ticket [2].
> DeleteRecordsCommand rewritten in java and moved to tools module.
> Currently, commands doesn’t have any tests so PR adds some tests, also.
>
> Please, review.
>
> [1] https://github.com/apache/kafka/pull/13278
> [2] https://issues.apache.org/jira/browse/KAFKA-14591
>

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

Hi Omnia and Chris, I agree with setting
"replication.policy.internal.topic.separator.enabled=true" by default,
but I was wondering if we should also deprecate and remove in Kafka 4.
If there is agreement in having the same behavior for internal and
non-internal topics, then it should be fine, and we won't need to keep
an additional configuration around. Wdyt?

Cheers
Fede

On Fri, Jul 14, 2023 at 1:51 PM Omnia Ibrahim  wrote:
>
> Hi Chris, I added a section for backport plan here 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy#KIP949:AddflagtoenabletheusageoftopicseparatorinMM2DefaultReplicationPolicy-Backportingplan
>
> Cheers,
> Omnia
>
> > On 13 Jul 2023, at 19:33, Chris Egerton  wrote:
> >
> > Hi Omnia,
> >
> > Yes, I think we ought to state the backport plan in the KIP, since it's
> > highly unusual for KIP changes or new configuration properties to be
> > backported and we should get the approval of the community (including
> > binding and non-binding voters) before implementing it.
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Jul 13, 2023 at 7:13 AM Omnia Ibrahim 
> > wrote:
> >
> >> Hi Chris,
> >> The implementation should be very small so backporting this to 3.1 and 3.2
> >> would be perfect for this case if you or any other committer are okay with
> >> approving the backporting. Do we need to state this in KIP as well or not?
> >>
> >> Also, I’ll open a vote for the KIP today and prepare the pr for it so we
> >> can merge it as soon as we can.
> >>
> >> Thanks,
> >>
> >> Omnia
> >>
> >> On Wed, Jul 12, 2023 at 4:31 PM Chris Egerton 
> >> wrote:
> >>
> >>> Hi Omnia,
> >>>
> >>> Thanks for changing the default, LGTM 
> >>>
> >>> As far as backporting goes, we probably won't be doing another release
> >> for
> >>> 3.1, and possibly not for 3.2 either; however, if we can make the
> >>> implementation focused enough (which I don't think would be too
> >> difficult,
> >>> but correct me if I'm wrong), then we can still backport through 3.1.
> >> Even
> >>> if we don't do another release it can make life easier for people who are
> >>> maintaining parallel forks. Obviously this shouldn't be taken as a
> >> blanket
> >>> precedent but in this case it seems like the benefits may outweigh the
> >>> costs. What are your thoughts?
> >>>
> >>> Cheers,
> >>>
> >>> Chris
> >>>
> >>> On Wed, Jul 12, 2023 at 9:05 AM Omnia Ibrahim 
> >>> wrote:
> >>>
>  Hi Chris, thanks for the feedback.
>  1. regarding the default value I had the same conflict of which version
> >>> to
>  break the backward compatibility with. We can just say that this KIP
> >>> gives
>  the release Pre KIP-690 the ability to keep the old behaviour with one
>  config and keep the backwards compatibility from post-KIP-690 the same
> >> so
>  we don't break at least the last 3 versions. I will update the KIP to
>  switch the default value to true.
>  2. For the backporting, which versions can we backport these to?
> >> Usually,
>  Kafka supports bugfix releases as needed for the last 3 releases. Now
> >> we
> >>> @
>  3.5 so the last 3 are 3.4, 3.3 and 3.2 is this correct?
>  3. I'll add a Jira for updating the docs for this KIP so we don't
> >> forget
>  about it.
> 
>  Thanks
>  Omnia
> 
> 
>  On Mon, Jul 10, 2023 at 5:33 PM Chris Egerton  >>>
>  wrote:
> 
> > Hi Omnia,
> >
> > Thanks for taking this on! I have some thoughts but the general
> >>> approach
> > looks good.
> >
> > 1. Default value
> >
> > One thing I'm wrestling with is what the default value of the new
>  property
> > should be. I know on the Jira ticket I proposed that it should be
> >>> false,
> > but I'm having second thoughts. Technically we'd preserve backward
> > compatibility with pre-KIP-690 releases by defaulting to false, but
> >> at
>  the
> > same time, we'd break compatibility with post-KIP-690 releases. And
> >> if
> >>> we
> > default to true, the opposite would be true: compatibility would be
>  broken
> > with pre-KIP-690 releases, but preserved with post-KIP-690 releases.
> >
> > One argument against defaulting to false (which, again, would
> >> preserve
>  the
> > behavior of MM2 before we accidentally broke compatibility with
> >>> KIP-690)
>  is
> > that this change could possibly cause a single MM2 setup to break
> > twice--once when upgrading from a pre-KIP-690 release to an existing
> > release, and again when upgrading from that existing release to a
> >>> version
> > that reverted (by default) to pre-KIP-690 behavior. On the other
> >> hand,
> >>> if
> > we default to true (which would preserve the existing behavior that
>  breaks
> > compatibility with pre-KIP-690 releases), then any given setup will
> >>> only
>  be
> > broken once.
> >
> > In 

Re: [VOTE] 3.5.1 RC1

[Federico Valeri]

Hi Divij, I did the following checks:

- Checked signature, checksum, licenses
- Spot checked documentation and javadoc
- Built from source with Java 17 and Scala 2.13
- Ran full unit and integration test suites
- Ran test Java app using staging Maven artifacts

+1 (non binding)

Cheers
Fede

On Mon, Jul 17, 2023 at 10:27 AM Divij Vaidya  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate (RC1) for release of Apache Kafka 3.5.1. First
> release candidate (RC0) was discarded due to incorrect license files. They
> have been fixed since then.
>
> This release is a security patch release. It upgrades the dependency,
> snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
> can find more information about the CVE at Kafka CVE list
> .
>
> Additionally, this releases fixes a regression introduced in 3.3.0, which
> caused security.protocol configuration values to be restricted to upper
> case only. With this release, security.protocol values are
> case insensitive. See KAFKA-15053
>  for details.
>
> Release notes for the 3.5.1 release:
> https://home.apache.org/~divijv/kafka-3.5.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, July 20, 9am PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~divijv/kafka-3.5.1-rc1/
>
> Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> Javadoc:
> https://home.apache.org/~divijv/kafka-3.5.1-rc1/javadoc/
>
> Tag to be voted upon (off 3.5 branch) is the 3.5.1 tag:
> https://github.com/apache/kafka/releases/tag/3.5.1-rc1
>
> Documentation:
> https://kafka.apache.org/35/documentation.html
> Please note that documentation will be updated with upgrade notes (
> https://github.com/apache/kafka/commit/4c78fd64454e25e3536e8c7ed5725d3fbe944a49)
> after the release is complete.
>
> Protocol:
> https://kafka.apache.org/35/protocol.html
>
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/43/ (2 failures)
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/42/ (6 failures)
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/39/ (9 failures)
>
> In all 3 runs above, there are no common tests which are failing, which
> leads me to believe that they are flaky. I have also verified that
> unit/integration tests on my local machine successfully pass (JDK 17 +
> Scala 2.13)
>
> System tests:
> Not planning to run system tests since this is a patch release.
>
> Thank you.
>
> --
> Divij Vaidya
> Release Manager for Apache Kafka 3.5.1

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

Hi Chris, the KIP says it would be a subclass of DefaultReplicationPolicy
that overrides the ReplicationPolicy.offsetSyncsTopic and
ReplicationPolicy.checkpointsTopic. So, not much to maintain and it would
be more intuitive, as you say.

On Wed, Jul 19, 2023, 4:50 PM Chris Egerton  wrote:

> HI all,
>
> I'm not sure I understand the benefits of introducing a separate
> replication policy class, besides maybe being more readable/intuitive to
> users who would want to know when to use one or the other. It feels like
> we've swapped out a "fix the bug" property for an entire "fix the bug"
> class, and it still leaves the problem of graceful migration from legacy
> behavior to new behavior unsolved. It would also require us to consider
> whether any future changes we make to the DefaultReplicationPolicy class
> would have to be ported over to the LegacyReplicationPolicy class as well.
>
> Perhaps I'm missing something; are there other benefits of introducing a
> separate replication policy class?
>
> Cheers,
>
> Chris
>
> On Wed, Jul 19, 2023 at 5:45 AM Omnia Ibrahim 
> wrote:
>
> > Hi Federico,
> > I like the idea of implementing `LegacyReplicationPolicy` and avoiding
> bug
> > fixes properties. We can drop the idea of the property and just go ahead
> > with introducing the `LegacyReplicationPolicy` and any user upgrade from
> > pre-KIP-690 can use this policy instead of default and no impact will
> > happen to users upgrading from 3.x to post-KIP-949. We can mark
> > `LegacyReplicationPolicy` as deprecated later if we want (but not in 4.0
> as
> > this is very soon) and we can drop it entirely at some point.
> >
> > If there's an agreement on this approach I can upgrade the KIP.
> >
> > Cheers.
> > Omnia
> >
> > On Wed, Jul 19, 2023 at 8:52 AM Federico Valeri 
> > wrote:
> >
> > > Hi, one way to avoid the "fix the bug property" would be to provide
> > > and document an additional LegacyReplicationPolicy, but still keeping
> > > the current DefaultReplicationPolicy as replication.policy.class
> > > default value, which is basically one of the workarounds suggested in
> > > the KIP.
> > >
> > > On Tue, Jul 18, 2023 at 9:49 PM Chris Egerton  >
> > > wrote:
> > > >
> > > > Hi Federico/Omnia,
> > > >
> > > > Generally I like the idea of deprecating and eventually removing "fix
> > the
> > > > bug" properties like this, but 4.0 may be a bit soon. I'm also unsure
> > of
> > > > how we would instruct users who are relying on the property being set
> > to
> > > > "false" to migrate to a version of MM2 that doesn't support support
> it,
> > > > beyond simply implementing their own replication policy--at which
> > point,
> > > > would we really be doing anyone a favor by forcing them to fork the
> > > default
> > > > policy just to preserve a property we removed?
> > > >
> > > > I guess right now I'd rather play it safe and not immediately
> deprecate
> > > the
> > > > property. If we can find an easy migration path for users who are
> > relying
> > > > on it, then I'd be happy to deprecate and schedule for removal.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Jul 18, 2023 at 12:54 PM Omnia Ibrahim <
> > o.g.h.ibra...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Federico,
> > > > > I don't have any strong opinion one way or the other. The pro of
> > > > > deprecation is not adding more configs to MM2 as it has too many
> > > configs
> > > > > already. However, we need to think about old MM2 migrating their
> > > internal
> > > > > topics to 4.0 with less impact.
> > > > >
> > > > > @Chris what do you think?
> > > > >
> > > > > Cheers
> > > > > Omnia
> > > > >
> > > > > On Fri, Jul 14, 2023 at 2:38 PM Federico Valeri <
> > fedeval...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Omnia and Chris, I agree with setting
> > > > > > "replication.policy.internal.topic.separator.enabled=true" by
> > > default,
> > > > > > but I was wondering if we should also deprecate and remove in
> Kafka
> > > 4.
> > > > > > If there is agreeme

Re: [VOTE] 3.5.1 RC0

[Federico Valeri]

Hi Divij, thanks for running the release.

I found a couple of issues with licenses. We include
plexus-utils-3.3.1, but the LICENSE file has plexus-utils-3.3.0. The
other issue is that we have classgraph-4.8.138, but we don't include
this library for a few releases (this is non blocking but it would be
good to fix).

Br
Fede




On Wed, Jul 12, 2023 at 12:14 PM Divij Vaidya  wrote:
>
> + kafka-clie...@googlegroups.com
>
> --
> Divij Vaidya
>
>
>
> On Wed, Jul 12, 2023 at 12:03 PM Divij Vaidya 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.5.1.
> >
> > This release is a security patch release. It upgrades the dependency,
> > snappy-java, to a version which is not vulnerable to CVE-2023-34455. You
> > can find more information about the CVE at Kafka CVE list
> > .
> >
> > Additionally, this releases fixes a regression introduced in 3.3.0, which
> > caused security.protocol configuration values to be restricted to upper
> > case only. With this release, security.protocol values are
> > case insensitive. See KAFKA-15053
> >  for details.
> >
> > Release notes for the 3.5.1 release:
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, July 18, 9am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/
> >
> > Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > Javadoc:
> > https://home.apache.org/~divijv/kafka-3.5.1-rc0/javadoc/
> >
> > Tag to be voted upon (off 3.5 branch) is the 3.5.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.5.1-rc0
> >
> > Documentation:
> > https://kafka.apache.org/35/documentation.html
> > Please note that documentation will be updated with upgrade notes (
> > https://github.com/apache/kafka/commit/4c78fd64454e25e3536e8c7ed5725d3fbe944a49)
> > after the release is complete.
> >
> > Protocol:
> > https://kafka.apache.org/35/protocol.html
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/35/ (9
> > failures). I am running another couple of runs to ensure that there are no
> > consistently failing tests. I have verified that unit/integration tests on
> > my local machine successfully pass.
> >
> > System tests:
> > Not planning to run system tests since this is a patch release.
> >
> > Thank you.
> >
> > --
> > Divij Vaidya
> > Release Manager for Apache Kafka 3.5.1
> >
> >

Re: [VOTE] 3.4.1 RC3

[Federico Valeri]

Hi Luke,

- Checked signature, checksum, libs, and license file
- Build from source with Java 17 and Scala 2.13
- Ran full unit and integration test suites
- Ran Java app with staging Maven artifacts against ZK and KRaft
multi-node clusters

+1 (non binding)

Thanks
Fede


On Fri, May 26, 2023 at 1:42 PM Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks again for running this!
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
> -- For KRaft, I looked at the process running Kafka and confirmed that the
> spamming log message is not present anymore ("Generated a metadata delta
> between...")
> - Checked that the diff in LICENSE-binary is identical than previous RC
> -- For each difference of the previous one I checked the binary on the
> scala 2.13 package had the version specified in the file
>
> This gets a +1 from me.
>
> Best
>
> On Fri, May 26, 2023 at 9:33 AM Luke Chen  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the 4th candidate for release of Apache Kafka 3.4.1.
> >
> > This is a bugfix release with several fixes since the release of 3.4.0. A
> > few of the major issues include:
> > - core
> > KAFKA-14644  Process
> > should stop after failure in raft IO thread
> > KAFKA-14946  KRaft
> > controller node shutting down while renouncing leadership
> > KAFKA-14887  ZK session
> > timeout can cause broker to shutdown
> > - client
> > KAFKA-14639  Kafka
> > CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> > - connect
> > KAFKA-12558  MM2 may
> > not
> > sync partition offsets correctly
> > KAFKA-14666  MM2 should
> > translate consumer group offsets behind replication flow
> > - stream
> > KAFKA-14172  bug: State
> > stores lose state when tasks are reassigned under EOS
> >
> >
> > Release notes for the 3.4.1 release:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Jun 2, 2023
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/
> >
> > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.4.1-rc3
> >
> > * Documentation: (will be updated after released)
> > https://kafka.apache.org/34/documentation.html
> >
> > * Protocol: (will be updated after released)
> > https://kafka.apache.org/34/protocol.html
> >
> > The most recent build has had test failures. These all appear to be due to
> > flakiness, but it would be nice if someone more familiar with the failed
> > tests could confirm this. I may update this thread with passing build links
> > if I can get one, or start a new release vote thread if test failures must
> > be addressed beyond re-running builds until they pass.
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/141/
> >
> > System tests:
> > Will update the results later
> >
> > Thank you
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Adding non-committers as Github collaborators

[Federico Valeri]

+1 on Divij suggestions


On Wed, May 24, 2023 at 12:04 PM Divij Vaidya  wrote:
>
> Hey folks
>
> A week into this experiment, I am finding the ability to add labels,
> request for reviewers and ability to close PRs very useful.
>
> 1. May I suggest an improvement to the process by requesting for some
> guidance on the interest areas for various committers. This would help us
> request for reviews from the right set of individuals.
> As a reference, we have tried something similar with Apache TinkerPop (see
> TinkerPop Contributors section at the end) [1], where the committers self
> identify their preferred area of interest.
>
> 2. I would also request creation of the following new labels:
> tiered-storage, transactions, security, refactor, zk-migration,
> first-contribution (so that we can prioritize reviews for first time
> contributors as an encouragement), build, metrics
>
> [1] https://tinkerpop.apache.org/
>
> --
> Divij Vaidya
>
>
>
> On Mon, May 15, 2023 at 11:07 PM John Roesler  wrote:
>
> > Hello again, all,
> >
> > Just a quick update: after merging the changes to asf.yaml, I received a
> > notification that the list is limited to only 10 people, not 20 as the
> > documentation states.
> >
> > Here is the list of folks who will now be able to triage PRs and trigger
> > builds: Victoria Xia, Greg Harris, Divij Vaidya, Lucas Brutschy, Yash
> > Mayya, Philip Nee, vamossagar12, Christo Lolov, Federico Valeri, and andymg3
> >
> > Thanks all,
> > -John
> >
> > On 2023/05/12 15:53:40 John Roesler wrote:
> > > Thanks again for bringing this up, David!
> > >
> > > As an update to the community, the PMC has approved a process to make
> > use of this feature.
> > >
> > > Here are the relevant updates:
> > >
> > > PR to add the policy: https://github.com/apache/kafka-site/pull/510
> > >
> > > PR to update the list: https://github.com/apache/kafka/pull/13713
> > >
> > > Ticket to automate this process.. Contributions welcome :)
> > https://issues.apache.org/jira/browse/KAFKA-14995
> > >
> > > And to make sure it doesn't fall through the cracks in the mean time,
> > here's the release process step:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-UpdatetheCollaboratorsList
> > >
> > > Unfortunately, the "collaborator" feature only allows 20 usernames, so
> > we have decided to simply take the top 20 non-committer authors from the
> > past year (according to git shortlog). Congratulations to our new
> > collaborators!
> > >
> > > Victoria Xia, Greg Harris, Divij Vaidya, Lucas Brutschy, Yash Mayya,
> > Philip Nee, vamossagar12, Christo Lolov, Federico Valeri, and andymg3
> > >
> > > Thanks,
> > > -John
> > >
> > > On 2023/04/27 18:45:09 David Arthur wrote:
> > > > Hey folks,
> > > >
> > > > I stumbled across this wiki page from the infra team that describes the
> > > > various features supported in the ".asf.yaml" file:
> > > >
> > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
> > > >
> > > > One section that looked particularly interesting was
> > > >
> > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AssigningexternalcollaboratorswiththetriageroleonGitHub
> > > >
> > > > github:
> > > >   collaborators:
> > > > - userA
> > > > - userB
> > > >
> > > > This would allow us to define non-committers as collaborators on the
> > Github
> > > > project. Concretely, this means they would receive the "triage" Github
> > role
> > > > (defined here
> > > >
> > https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/repository-roles-for-an-organization#permissions-for-each-role
> > ).
> > > > Practically, this means we could let non-committers do things like
> > assign
> > > > labels and reviewers on Pull Requests.
> > > >
> > > > I wanted to see what the committer group thought about this feature. I
> > > > think it could be useful.
> > > >
> > > > Cheers,
> > > > David
> > > >
> > >
> >

Re: [VOTE] 3.4.1 RC2

[Federico Valeri]

Hi Luke, these are the things I checked:

- Source signature, checksum and license file
- Build from source with Java 17 and Scala 2.13
- Unit and integration test suites
- Java app with staging Maven artifacts

+1 (non binding)

Thanks
Fede


On Wed, May 24, 2023 at 6:36 PM Chris Egerton  wrote:
>
> Hi Luke,
>
> Thanks again for running this release! Echoing your hopes that this will be
> the final 3.4.1 release candidate :)
>
> To verify, I:
> - Built from source with Java 11 with both:
> - - the 3.4.1-rc2 tag on GitHub
> - - the kafka-3.4.1-src.tgz artifact from
> https://home.apache.org/~showuon/kafka-3.4.1-rc2/
> - Checked signatures and checksums
> - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> https://home.apache.org/~showuon/kafka-3.4.1-rc2/ with Java 11 and Scala 13
> in KRaft mode
> - Ran all unit tests
> - - No failures this time! 
> - Ran all integration tests for Connect and MM2
>
> +1 (binding)
>
> Cheers,
>
> Chris
>
> On Wed, May 24, 2023 at 6:12 AM Luke Chen  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the 3rd candidate for release of Apache Kafka 3.4.1.
> >
> > This is a bugfix release with several fixes since the release of 3.4.0. A
> > few of the major issues include:
> > - core
> > KAFKA-14644  Process
> > should stop after failure in raft IO thread
> > KAFKA-14946  KRaft
> > controller node shutting down while renouncing leadership
> > KAFKA-14887  ZK session
> > timeout can cause broker to shutdown
> > - client
> > KAFKA-14639  Kafka
> > CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> > - connect
> > KAFKA-12558  MM2 may
> > not
> > sync partition offsets correctly
> > KAFKA-14666  MM2 should
> > translate consumer group offsets behind replication flow
> > - stream
> > KAFKA-14172  bug: State
> > stores lose state when tasks are reassigned under EOS
> >
> >
> > Release notes for the 3.4.1 release:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by May 31, 2023
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~showuon/kafka-3.4.1-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc2/javadoc/
> >
> > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.4.1-rc2
> >
> > * Documentation: (will be updated after released)
> > https://kafka.apache.org/34/documentation.html
> >
> > * Protocol: (will be updated after released)
> > https://kafka.apache.org/34/protocol.html
> > The most recent build has had test failures. These all appear to be due to
> > flakiness, but it would be nice if someone more familiar with the failed
> > tests could confirm this. I may update this thread with passing build links
> > if I can get one, or start a new release vote thread if test failures must
> > be addressed beyond re-running builds until they pass.
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/138/
> >
> > System tests:
> > Will update the results later
> >
> > Hope this will be the final RC for v3.4.1!
> >
> > Thank you.
> > Luke
> >

Re: [VOTE] 3.5.0 RC1

[Federico Valeri]

Hi Mickael, I did the following checks:

- Signature, checksum, licenses
- Build from source with Java 17 and Scala 2.13
- Full unit and integration test suites
- Java app with staging Maven artifacts and multi-node cluster

LGTM, apart from the already mentioned "classgraph" license that can be removed.

+1 (non binding)

Thanks
Fede

On Tue, Jun 6, 2023 at 9:39 AM Luke Chen  wrote:
>
> Hi Mickael,
>
> I ran the following validation steps:
> - Built from source with Java 17 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the quickstart in KRaft and Zookeeper mode
>
> +1 (binding) from me.
> Thanks for running the release.
>
> Luke
>
> On Tue, Jun 6, 2023 at 4:23 AM Josep Prat 
> wrote:
>
> > Hi Mickael,
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> > -- For KRaft, I looked at the process running Kafka and confirmed that the
> > spamming log message is not present anymore ("Generated a metadata delta
> > between...")
> > - Checked the contents of LICENSE-binary against the lib folder
> > -- I found that the LICENSE-binary has a reference to classgraph-4.8.138
> > but I fail to find this library in the lib folder. Trying to backtrack when
> > it was added, it seems it was done here:
> >
> > https://github.com/apache/kafka/commit/3a2ac267178c5464e41b94fcbb2dd897212812bd
> > but I fail to find this library in the 3.3.1 lib folder (3.3.0 was a dead
> > release).
> >
> > I'm not sure this qualifies as a blocker for the release though, I leave it
> > up to you.
> >
> > Best,
> >
> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 3.5.0. Some
> > > of the major features include:
> > > - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > > 2.0 clusters
> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > - KIP-887: Add ConfigProvider to make use of environment variables
> > > - KIP-889: Versioned State Stores
> > > - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > > Kafka Brokers
> > >
> > > Release notes for the 3.5.0 release:
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/35/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/35/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.5 branch:
> > > Unit/integration tests: I'm struggling to get all tests to pass in the
> > > same build. I'll run a few more builds to ensure each test pass at
> > > least once in the CI. All tests passed locally.
> > > System tests: The build is still running, I'll send an update once I
> > > have the results.
> > >
> > > Thanks,
> > > Mickael
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |    > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >

Re: [ANNOUNCE] Apache Kafka 3.4.1

[Federico Valeri]

Thanks Luke!

On Wed, Jun 7, 2023 at 5:56 AM Kamal Chandraprakash
 wrote:
>
> Thanks Luke for running this release!
>
> On Wed, Jun 7, 2023 at 8:08 AM Chia-Ping Tsai  wrote:
>
> > Thank Luke for this hard work!!!
> >
> > > Chris Egerton  於 2023年6月7日 上午10:35 寫道：
> > >
> > > Thanks for running this release, Luke!
> > >
> > > On Tue, Jun 6, 2023, 22:31 Luke Chen  wrote:
> > >
> > >> The Apache Kafka community is pleased to announce the release for
> > >> Apache Kafka 3.4.1.
> > >>
> > >> This is a bug fix release and it includes fixes and improvements from
> > >> 58 JIRAs, including a few critical bugs:
> > >> - core
> > >> KAFKA-14644 Process should stop after failure in raft IO thread
> > >> KAFKA-14946 KRaft controller node shutting down while renouncing
> > leadership
> > >> KAFKA-14887 ZK session timeout can cause broker to shutdown
> > >> - client
> > >> KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
> > >> in one rebalance cycle
> > >> - connect
> > >> KAFKA-12558 MM2 may not sync partition offsets correctly
> > >> KAFKA-14666 MM2 should translate consumer group offsets behind
> > replication
> > >> flow
> > >> - stream
> > >> KAFKA-14172 bug: State stores lose state when tasks are reassigned under
> > >> EOS
> > >>
> > >> All of the changes in this release can be found in the release notes:
> > >>
> > >> https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html
> > >>
> > >> You can download the source and binary release (Scala 2.12 and Scala
> > 2.13)
> > >> from:
> > >>
> > >> https://kafka.apache.org/downloads#3.4.1
> > >>
> > >>
> > >>
> > ---
> > >>
> > >> Apache Kafka is a distributed streaming platform with four core APIs:
> > >>
> > >> ** The Producer API allows an application to publish a stream records
> > >> to one or more Kafka topics.
> > >>
> > >> ** The Consumer API allows an application to subscribe to one or more
> > >> topics and process the stream of records produced to them.
> > >>
> > >> ** The Streams API allows an application to act as a stream processor,
> > >> consuming an input stream from one or more topics and producing an
> > >> output stream to one or more output topics, effectively transforming
> > >> the input streams to output streams.
> > >>
> > >> ** The Connector API allows building and running reusable producers or
> > >> consumers that connect Kafka topics to existing applications or data
> > >> systems. For example, a connector to a relational database might
> > >> capture every change to a table.
> > >>
> > >>
> > >> With these APIs, Kafka can be used for two broad classes of application:
> > >>
> > >> ** Building real-time streaming data pipelines that reliably get data
> > >> between systems or applications.
> > >>
> > >> ** Building real-time streaming applications that transform or react
> > >> to the streams of data.
> > >>
> > >> Apache Kafka is in use at large and small companies worldwide,
> > >> including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
> > >> Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
> > >> Zalando, among others.
> > >>
> > >> A big thank you for the following 32 contributors to this release!
> > >>
> > >> atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
> > >> csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
> > >> emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
> > >> hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
> > >> Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> > >> Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
> > >> Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
> > >> Mayya
> > >>
> > >> We welcome your help and feedback. For more information on how to
> > >> report problems, and to get involved, visit the project website at
> > >> https://kafka.apache.org/
> > >>
> > >>
> > >> Thank you!
> > >>
> > >> Regards,
> > >> Luke
> > >>
> >
> >

Re: [VOTE] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

The KIP is accepted with 3 binding votes (Luke, Ziming, John) and 2
non binding votes (Divij, Kamal).

Thanks everyone!

Cheers
Fede

On Sat, May 20, 2023 at 10:58 PM John Roesler  wrote:
>
> I’m +1 (binding)
>
> Thanks, Federico!
>
> It looks like a nice improvement to me.
>
> -John
>
>
> On Sat, May 20, 2023, at 09:16, Kamal Chandraprakash wrote:
> > +1 (non binding)
> >
> > On Wed, May 17, 2023 at 3:22 AM Divij Vaidya 
> > wrote:
> >
> >> +1 (non binding)
> >>
> >> Divij Vaidya
> >>
> >>
> >>
> >> On Tue, May 16, 2023 at 4:35 AM ziming deng 
> >> wrote:
> >>
> >> > Thanks for this improvement, +1 from me(binging)
> >> >
> >> > —
> >> > Best,
> >> > Ziming
> >> >
> >> > > On May 16, 2023, at 00:43, Federico Valeri 
> >> wrote:
> >> > >
> >> > > Hi all,
> >> > >
> >> > > I'd like to start a vote on KIP-927: Improve the kafka-metadata-quorum
> >> > output.
> >> > >
> >> > >
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> >> > >
> >> > > Discussion thread:
> >> > > https://lists.apache.org/thread/pph59hxvz5jkk709x53p44xrpdqwv8qc
> >> > >
> >> > > Thanks
> >> > > Fede
> >> >
> >> >
> >>

Re: [VOTE] 3.4.1 RC1

[Federico Valeri]

Hi Luke,

- Source signature and checksum
- Build from source with Java 17 and Scala 2.13
- Full unit and integration test suite
- Java app with staging Maven artifacts

+1 (non binding)

Thanks
Fede

PS: Links still point to RC0, but I checked and RC1 artifacts are
there, including Maven. The only risk I see is that you may actually
test with the wrong artifacts. To avoid any confusion, I would suggest
to resend them on this thread.






On Mon, May 22, 2023 at 2:53 PM Luke Chen  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the 2nd candidate for release of Apache Kafka 3.4.1.
>
> This is a bugfix release with several fixes since the release of 3.4.0. A
> few of the major issues include:
> - core
> KAFKA-14644  Process
> should stop after failure in raft IO thread
> KAFKA-14946  KRaft
> controller node shutting down while renouncing leadership
> KAFKA-14887  ZK session
> timeout can cause broker to shutdown
> - client
> KAFKA-14639  Kafka
> CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> - connect
> KAFKA-12558  MM2 may not
> sync partition offsets correctly
> KAFKA-14666  MM2 should
> translate consumer group offsets behind replication flow
> - stream
> KAFKA-14172  bug: State
> stores lose state when tasks are reassigned under EOS
>
> Release notes for the 3.4.1 release:
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by May 29, 2023
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/
>
> * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> https://github.com/apache/kafka/releases/tag/3.4.1-rc0
>
> * Documentation: (will be updated after released)
> https://kafka.apache.org/34/documentation.html
>
> * Protocol: (will be updated after released)
> https://kafka.apache.org/34/protocol.html
>
> The most recent build has had test failures. These all appear to be due to
> flakiness, but it would be nice if someone more familiar with the failed
> tests could confirm this. I may update this thread with passing build links
> if I can get one, or start a new release vote thread if test failures must
> be addressed beyond re-running builds until they pass.
>
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/135/
> 
>
> System tests:
> Will update the results later
>
> Confirmed Maven artifacts are in staging repository.
>
> Thank you.
> Luke

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

+1 (non binding)

Thanks
Fede


On Sun, Jul 23, 2023 at 6:30 PM Omnia Ibrahim  wrote:
>
> Hi everyone,
> I would like to open a vote for KIP-949. The proposal is here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> .
> 
>
> Thanks
> Omnia

Re: [ANNOUNCE] Apache Kafka 3.5.1

[Federico Valeri]

Thanks Divij :)

On Mon, Jul 24, 2023 at 2:28 PM Luke Chen  wrote:
>
> Thanks for running the release!
>
> Luke
>
> On Mon, Jul 24, 2023 at 6:35 PM Manyanda Chitimbo <
> manyanda.chiti...@gmail.com> wrote:
>
> > Thanks for running the release Divij.
> >
> > On Mon 24 Jul 2023 at 12:15, Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Thanks for running the release, Divij!
> > >
> > > On Mon, Jul 24, 2023 at 3:17 PM Divij Vaidya  wrote:
> > >
> > > > The Apache Kafka community is pleased to announce the release for
> > Apache
> > > > Kafka 3.5.1
> > > >
> > > > This release is a security patch release. It upgrades the dependency,
> > > > snappy-java, to a version which is not vulnerable to CVE-2023-34455.
> > You
> > > > can find more information about the CVE at Kafka CVE list
> > > > .
> > > >
> > > > Additionally, this releases fixes a regression introduced in 3.3.0,
> > which
> > > > caused security.protocol configuration values to be restricted to upper
> > > > case only. With this release, security.protocol values are
> > > > case insensitive. See KAFKA-15053
> > > >  for details.
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://www.apache.org/dist/kafka/3.5.1/RELEASE_NOTES.html
> > > >
> > > > You can download the source and binary release (Scala 2.11 and Scala
> > > 2.12)
> > > > from:
> > > > https://kafka.apache.org/downloads#3.5.1
> > > >
> > > >
> > > >
> > >
> > ---
> > > >
> > > > Apache Kafka is a distributed streaming platform with four core APIs:
> > > >
> > > >
> > > > ** The Producer API allows an application to publish a stream of
> > records
> > > to
> > > > one or more Kafka topics.
> > > >
> > > > ** The Consumer API allows an application to subscribe to one or more
> > > > topics and process the stream of records produced to them.
> > > >
> > > > ** The Streams API allows an application to act as a stream processor,
> > > > consuming an input stream from one or more topics and producing an
> > > > output stream to one or more output topics, effectively transforming
> > the
> > > > input streams to output streams.
> > > >
> > > > ** The Connector API allows building and running reusable producers or
> > > > consumers that connect Kafka topics to existing applications or data
> > > > systems. For example, a connector to a relational database might
> > > > capture every change to a table.
> > > >
> > > > With these APIs, Kafka can be used for two broad classes of
> > application:
> > > >
> > > > ** Building real-time streaming data pipelines that reliably get data
> > > > between systems or applications.
> > > >
> > > > ** Building real-time streaming applications that transform or react
> > > > to the streams of data.
> > > >
> > > > Apache Kafka is in use at large and small companies worldwide,
> > including
> > > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> > Rabobank,
> > > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > > >
> > > > A big thank you for the following 22 contributors to this release!
> > > (Please
> > > > report an unintended omission)
> > > >
> > > > Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai,
> > > > Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij
> > > > Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael
> > Maison,
> > > > minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya
> > > >
> > > > We welcome your help and feedback. For more information on how to
> > > > report problems, and to get involved, visit the project website at
> > > > https://kafka.apache.org/
> > > >
> > > > Thank you!
> > > >
> > > > Regards,
> > > > Divij Vaidya
> > > > Release Manager for Apache Kafka 3.5.1
> > > >
> > >
> > --
> > Manyanda Chitimbo.
> >

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Federico Valeri]

Hi, the point that the legacy approach can only be taken once is
valid, so LGTM. Thanks.

Cheers
Fede

On Fri, Jul 21, 2023 at 4:28 PM Chris Egerton  wrote:
>
> Hi Omnia,
>
> LGTM, thanks! We may want to note the LegacyReplicationPolicy option in the
> rejected alternatives section in case others prefer that option.
>
> Given that we'd like this to land in time for 3.6.0, you may want to start
> a vote thread soon.
>
> Cheers,
>
> Chris
>
> On Fri, Jul 21, 2023 at 10:08 AM Omnia Ibrahim 
> wrote:
>
> > Hi Chris and Federico,
> > thinking about I think Chris's concern is valid and the bigger concern here
> > is that having this `LegacyReplicationPolicy` will eventually open the door
> > for diversion at some point between this `LegacyReplicationPolicy` and the
> > default one.
> > For now, let's have the flag properly fix this bug and we can keep it as an
> > option for people to switch between both behaviours. I know having a
> > bug-fix property is not great but we can treat it as a backward
> > compatibility property instead in order to keep old mirrors using the old
> > internal topics.
> >
> > Hope this is reasonable for the time being.
> >
> > Cheers,
> > Omnia
> >
> > On Wed, Jul 19, 2023 at 11:16 PM Chris Egerton 
> > wrote:
> >
> > > Hi Federico,
> > >
> > > Ah yes, sorry about that! You're correct that this would keep the two
> > > classes in line and largely eliminate the concern I posed about porting
> > > changes to both. Still, I'm a bit hesitant, and I'm not actually certain
> > > that this alternative is more intuitive. The name isn't very descriptive,
> > > and this is the kind of approach we can only really take once; if we
> > break
> > > compatibility again, would we introduce a LegacyLegacyReplicationPolicy?
> > > LegacyReplicationPolicy2? Finally, it seems a bit strange to introduce a
> > > new class to implement a change in behavior this small.
> > >
> > > That said, I don't think this is worth blocking on and wouldn't be
> > opposed
> > > if others felt strongly that a new replication policy class is superior
> > to
> > > a new property on the existing class.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Jul 19, 2023 at 2:53 PM Federico Valeri 
> > > wrote:
> > >
> > > > Hi Chris, the KIP says it would be a subclass of
> > DefaultReplicationPolicy
> > > > that overrides the ReplicationPolicy.offsetSyncsTopic and
> > > > ReplicationPolicy.checkpointsTopic. So, not much to maintain and it
> > would
> > > > be more intuitive, as you say.
> > > >
> > > > On Wed, Jul 19, 2023, 4:50 PM Chris Egerton 
> > > > wrote:
> > > >
> > > > > HI all,
> > > > >
> > > > > I'm not sure I understand the benefits of introducing a separate
> > > > > replication policy class, besides maybe being more readable/intuitive
> > > to
> > > > > users who would want to know when to use one or the other. It feels
> > > like
> > > > > we've swapped out a "fix the bug" property for an entire "fix the
> > bug"
> > > > > class, and it still leaves the problem of graceful migration from
> > > legacy
> > > > > behavior to new behavior unsolved. It would also require us to
> > consider
> > > > > whether any future changes we make to the DefaultReplicationPolicy
> > > class
> > > > > would have to be ported over to the LegacyReplicationPolicy class as
> > > > well.
> > > > >
> > > > > Perhaps I'm missing something; are there other benefits of
> > introducing
> > > a
> > > > > separate replication policy class?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, Jul 19, 2023 at 5:45 AM Omnia Ibrahim <
> > o.g.h.ibra...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Federico,
> > > > > > I like the idea of implementing `LegacyReplicationPolicy` and
> > > avoiding
> > > > > bug
> > > > > > fixes properties. We can drop the idea of the property and just go
> > > > ahead
> > > > > > with introducing the `LegacyReplicationPolicy` and any user upgrade
> > > &

Re: [ANNOUNCE] New committer: Divij Vaidya

[Federico Valeri]

Congrats Divij!

On Wed, Jun 14, 2023 at 10:02 AM Divij Vaidya  wrote:
>
> Thank you everyone! I promise to shoulder this new responsibility to the
> best of my abilities.
>
> --
> Divij Vaidya
>
>
>
> On Wed, Jun 14, 2023 at 4:13 AM ziming deng 
> wrote:
>
> > Congratulations, Divij!
> > Well deserved!
> >
> > --
> > Ziming
> >
> > > On Jun 14, 2023, at 09:41, Luke Chen  wrote:
> > >
> > > Congratulations, Divij!
> > > Well deserved!
> > >
> > > Luke
> > >
> > > On Wed, Jun 14, 2023 at 7:01 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > >> Congrats Divij!
> > >>
> > >> On Tue, Jun 13, 2023, 20:27 Philip Nee  wrote:
> > >>
> > >>> Congrats!
> > >>>
> > >>> On Tue, Jun 13, 2023 at 8:17 PM Randall Hauch 
> > wrote:
> > >>>
> >  Congratulations!
> > 
> >  On Tue, Jun 13, 2023 at 12:48 PM Matthias J. Sax 
> > >>> wrote:
> > 
> > > Congrats!
> > >
> > > On 6/13/23 10:24 AM, Satish Duggana wrote:
> > >> Congratulations Divij!!
> > >>
> > >> On Tue, 13 Jun 2023 at 22:41, Manyanda Chitimbo
> > >>  wrote:
> > >>>
> > >>> Congratulations Divij.
> > >>>
> > >>> On Tue 13 Jun 2023 at 17:50, Bruno Cadonna 
> >  wrote:
> > >>>
> >  Hi all,
> > 
> >  The PMC of Apache Kafka is pleased to announce a new Kafka
> > >>> committer
> >  Divij Vaidya.
> > 
> >  Divij's major contributions are:
> > 
> >  GDPR compliance enforcement of kafka-site -
> >  https://issues.apache.org/jira/browse/KAFKA-13868
> > 
> >  Performance improvements:
> > 
> >  Improve performance of VarInt encoding and decoding -
> >  https://github.com/apache/kafka/pull/13312
> > 
> >  Reduce data copy & buffer allocation during decompression -
> >  https://github.com/apache/kafka/pull/13135
> > 
> >  He also was heavily involved in the migration to Mockito.
> > 
> >  Furthermore, Divij is very active on the mailing lists as well as
> > >>> in
> >  maintaining and reviewing pull requests.
> > 
> >  Congratulations, Divij!
> > 
> >  Thanks,
> > 
> >  Bruno (on behalf of the Apache Kafka PMC)
> > 
> > 
> >  --
> > >>> Manyanda Chitimbo.
> > >
> > 
> > >>>
> > >>
> >
> >

[DISCUSS] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

Hi all, I'd like to start a new discussion thread on KIP-927: Improve
the kafka-metadata-quorum output.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output

This KIP is small and proposes to add a new optional flag to have a
human-readable timestamp output.

Thank you!

Br
Fede

Re: [DISCUSS] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

Hi Divij, thanks for having a look.

On Wed, May 10, 2023 at 8:03 PM Divij Vaidya  wrote:
>
> Thank you for the KIP.
>
> The current proposal has the limitation that it uses a duration syntax for
> representation of a timestamp. Also, the syntax relies on the locale and
> timezone of the caller machine. This makes it difficult to share the output
> with others. As an example, let's say you want to share the state of the
> quorum with me which was captured but "3s ago" gives me no information on
> when it was executed.
>

AFAIK, Unix timestamps are based on UTC, so there is no locale involved here.

> Alternatively, may I suggest introducing a parameter called
> "--datetime-format=" which takes as value the ISO-8601 [1] format and
> prints the timestamp based on the provided format. It would solve the
> problem of readability (epochs are hard to read!) as well as the problem of
> portability of output across machines.
>
> What do you think?
>

This could be a further improvement, but my point is that it is more
important to show the delay since the last time the operation was
performed, rather than the exact date. In that sense I think it is
more readable. Happy to hear other opinions on this.

> [1] https://en.wikipedia.org/wiki/ISO_8601
>
> --
> Divij Vaidya
>
>
>
> On Wed, May 10, 2023 at 6:10 PM Federico Valeri 
> wrote:
>
> > Hi all, I'd like to start a new discussion thread on KIP-927: Improve
> > the kafka-metadata-quorum output.
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> >
> > This KIP is small and proposes to add a new optional flag to have a
> > human-readable timestamp output.
> >
> > Thank you!
> >
> > Br
> > Fede
> >

Re: [DISCUSS] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

Hi Divij, I added this clarification and will start a vote soon.

Thanks.

On Mon, May 15, 2023 at 5:24 PM Divij Vaidya  wrote:
>
> Thanks for providing answers to my suggestions folks.
>
> Based on your answers, it seems like the KIP wants to do, relative duration
> = (now - timestamp), where "now" is in UTC (and not in your machine's
> locale/timezone). Perhaps, it's worth clarifying in the KIP?
> Other than that, no more comments from my side. The proposal looks good to
> me.
>
> --
> Divij Vaidya
>
>
>
> On Thu, May 11, 2023 at 9:52 AM Federico Valeri 
> wrote:
>
> > Hi Divij, thanks for having a look.
> >
> > On Wed, May 10, 2023 at 8:03 PM Divij Vaidya 
> > wrote:
> > >
> > > Thank you for the KIP.
> > >
> > > The current proposal has the limitation that it uses a duration syntax
> > for
> > > representation of a timestamp. Also, the syntax relies on the locale and
> > > timezone of the caller machine. This makes it difficult to share the
> > output
> > > with others. As an example, let's say you want to share the state of the
> > > quorum with me which was captured but "3s ago" gives me no information on
> > > when it was executed.
> > >
> >
> > AFAIK, Unix timestamps are based on UTC, so there is no locale involved
> > here.
> >
> > > Alternatively, may I suggest introducing a parameter called
> > > "--datetime-format=" which takes as value the ISO-8601 [1] format and
> > > prints the timestamp based on the provided format. It would solve the
> > > problem of readability (epochs are hard to read!) as well as the problem
> > of
> > > portability of output across machines.
> > >
> > > What do you think?
> > >
> >
> > This could be a further improvement, but my point is that it is more
> > important to show the delay since the last time the operation was
> > performed, rather than the exact date. In that sense I think it is
> > more readable. Happy to hear other opinions on this.
> >
> > > [1] https://en.wikipedia.org/wiki/ISO_8601
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Wed, May 10, 2023 at 6:10 PM Federico Valeri 
> > > wrote:
> > >
> > > > Hi all, I'd like to start a new discussion thread on KIP-927: Improve
> > > > the kafka-metadata-quorum output.
> > > >
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> > > >
> > > > This KIP is small and proposes to add a new optional flag to have a
> > > > human-readable timestamp output.
> > > >
> > > > Thank you!
> > > >
> > > > Br
> > > > Fede
> > > >
> >

[VOTE] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

Hi all,

I'd like to start a vote on KIP-927: Improve the kafka-metadata-quorum output.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output

Discussion thread:
https://lists.apache.org/thread/pph59hxvz5jkk709x53p44xrpdqwv8qc

Thanks
Fede

Re: [VOTE] 3.4.1 RC0

[Federico Valeri]

Hi Luke, thanks for running the release.

Looks like the Maven artifacts are not in staging:
https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/

Documentation still has 3.4.0, instead of 3.4.1 (not sure if this will
be aligned later):
https://kafka.apache.org/34/documentation.html#producerapi

Br
Fede


On Wed, May 17, 2023 at 5:24 AM Luke Chen  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for release of Apache Kafka 3.4.1.
>
> This is a bugfix release with several fixes since the release of 3.4.0. A
> few of the major issues include:
> - core
> KAFKA-14644  Process
> should stop after failure in raft IO thread
> KAFKA-14946  KRaft
> controller node shutting down while renouncing leadership
> KAFKA-14887  ZK session
> timeout can cause broker to shutdown
> - client
> KAFKA-14639  Kafka
> CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> - connect
> KAFKA-12558  MM2 may not
> sync partition offsets correctly
> KAFKA-14666  MM2 should
> translate consumer group offsets behind replication flow
> - stream
> KAFKA-14172  bug: State
> stores lose state when tasks are reassigned under EOS
>
> Release notes for the 3.4.1 release:
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by May 24, 2023
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/
>
> * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> https://github.com/apache/kafka/releases/tag/3.4.1-rc0
>
> * Documentation:
> https://kafka.apache.org/34/documentation.html
>
> * Protocol:
> https://kafka.apache.org/34/protocol.html
>
> The most recent build has had test failures. These all appear to be due to
> flakiness, but it would be nice if someone more familiar with the failed
> tests could confirm this. I may update this thread with passing build links
> if I can get one, or start a new release vote thread if test failures must
> be addressed beyond re-running builds until they pass.
>
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/133/
>
> System tests:
> Will update the results later
>
> Thank you.
> Luke

Re: [VOTE] 3.7.0 RC4

[Federico Valeri]

Hi Stanislav, it's +1 (non binding) for me.

- Built from source using Java 17 and Scala 2.13.
- Ran all unit and integration tests.
- Used the staged Maven artifacts to test some client applications
against ZK and KRaft based clusters.
- Tested the Cluster-wide dynamic log adjustment for Kafka Connect
with a 3 nodes cluster.
- Tried the new Docker image.
- Ran system tests (most failed tests are related to streams, see report below).

Thanks
Fede

---

### flaky

[INFO:2024-02-16 00:47:39,263]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'reassign_partitions_test.py', 'cls_name':
'ReassignPartitionsTest', 'method_name': 'test_reassign_partitions',
'injected_args': {'bounce_brokers': False, 'metadata_quorum':
'ISOLATED_KRAFT', 'reassign_from_offset_zero': False,
'use_new_coordinator': False}}

### failed (single tests)

[INFO:2024-02-15 23:18:38,332]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'downgrade_test.py', 'cls_name': 'TestDowngrade',
'method_name': 'test_upgrade_and_downgrade', 'injected_args':
{'compression_types': 'none', 'static_membership': True, 'version':
'2.8.2'}}

[INFO:2024-02-15 23:30:22,380]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/core',
'file_name': 'network_degrade_test.py', 'cls_name':
'NetworkDegradeTest', 'method_name': 'test_rate', 'injected_args':
{'device_name': 'eth0', 'latency_ms': 50, 'rate_limit_kbit': 100,
'task_name': 'rate-1000-latency-50'}}

[INFO:2024-02-16 05:44:00,934]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name': 'test_version_probing_upgrade',
'injected_args': None}

# failed (14 tests with increasing from_version)

[INFO:2024-02-16 05:14:15,576]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_application_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name': 'test_app_upgrade',
'injected_args': {'bounce_type': 'full', 'from_version': '2.2.2',
'to_version': '3.7.0-SNAPSHOT'}}

# failed (18 tests with increasing from_version)

[INFO:2024-02-16 05:37:08,823]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name':
'test_rolling_upgrade_with_2_bounces', 'injected_args':
{'from_version': '0.10.0.1', 'to_version': '3.7.0-SNAPSHOT'}}

# failed (4 tests with increasing from_version)

[INFO:2024-02-16 05:41:10,539]: RunnerClient: Loading test
{'directory': '/opt/kafka-dev/tests/kafkatest/tests/streams',
'file_name': 'streams_upgrade_test.py', 'cls_name':
'StreamsUpgradeTest', 'method_name':
'test_upgrade_downgrade_state_updater', 'injected_args':
{'from_version': '3.2.3', 'to_version': '3.7.0-SNAPSHOT', 'upgrade':
False}}








On Thu, Feb 15, 2024 at 3:39 PM Andrew Schofield
 wrote:
>
> +1 (non-binding). I used the staged binaries with Scala 2.13. I tried the new 
> group coordinator
> and consumer group protocol which is included with the Early Access release 
> of KIP-848.
> Also verified the availability of the new APIs. All working as expected.
>
> Thanks,
> Andrew
>
> > On 15 Feb 2024, at 05:07, Paolo Patierno  wrote:
> >
> > +1 (non-binding). I used the staged binaries with Scala 2.13 and mostly
> > focused on the ZooKeeper to KRaft migration with multiple tests. Everything
> > works fine.
> >
> > Thanks
> > Paolo
> >
> > On Mon, 12 Feb 2024, 22:06 Jakub Scholz,  wrote:
> >
> >> +1 (non-binding). I used the staged binaries with Scala 2.13 and the staged
> >> Maven artifacts to run my tests. All seems to work fine. Thanks.
> >>
> >> Jakub
> >>
> >> On Fri, Feb 9, 2024 at 4:20 PM Stanislav Kozlovski
> >>  wrote:
> >>
> >>> Hello Kafka users, developers and client-developers,
> >>>
> >>> This is the second candidate we are considering for release of Apache
> >> Kafka
> >>> 3.7.0.
> >>>
> >>> Major changes include:
> >>> - Early Access to KIP-848 - the next generation of the consumer rebalance
> >>> protocol
> >>> - Early Access to KIP-858: Adding JBOD support to KRaft
> >>> - KIP-714: Observability into Client metrics via a standardized interface
> >>>
> >>> Release notes for the 3.7.0 release:
> >>>
> >>>
> >> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/RELEASE_NOTES.html
> >>>
> >>> *** Please download, test and vote by Thursday, February 15th, 9AM PST
> >> ***
> >>>
> >>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>> https://kafka.apache.org/KEYS
> >>>
> >>> * Release artifacts to be voted upon (source and binary):
> >>> https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc4/
> >>>
> >>> * Docker release artifact to be voted upon:
> >>> apache/kafka:3.7.0-rc4
> >>>
> >>> * Maven artifacts to be voted upon:
> >>> 

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Federico Valeri]

HI Ziming, thanks for the KIP. Looks good to me.

Just on question: given that alterConfig is deprecated, shouldn't we
also introduce --disable-incremental as deprecated? That way we would
get rid of both in Kafka 4.0. Also see:
https://issues.apache.org/jira/browse/KAFKA-14705.

On Tue, Dec 19, 2023 at 9:05 AM ziming deng  wrote:
>
> Thank you for mention this Ismael,
>
> I added this to the motivation section, and I think we can still update 
> configs in this case by passing all sensitive configs, which is weird and not 
> friendly.
>
> --
> Best,
> Ziming
>
> > On Dec 19, 2023, at 14:24, Ismael Juma  wrote:
> >
> > Thanks for the KIP. I think one of the main benefits of the change isn't 
> > listed: sensitive configs make it impossible to make updates with the 
> > current cli tool because sensitive config values are never returned.
> >
> > Ismael
> >
> > On Mon, Dec 18, 2023 at 7:58 PM ziming deng  > > wrote:
> >>
> >> Hello, I want to start a discussion on KIP-1011, to make the broker config 
> >> change path unified with that of user/topic/client-metrics and avoid some 
> >> bugs.
> >>
> >> Here is the link:
> >>
> >> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
> >> kafka-configs.sh - Apache Kafka - Apache Software Foundation
> >> cwiki.apache.org
> >>
> >>  
> >> KIP-1011:
> >>  Use incrementalAlterConfigs when updating broker configs by 
> >> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
> >> 
> >> cwiki.apache.org 
> >> 
> >>  
> >> 
> >>
> >> Best,
> >> Ziming.
>

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Federico Valeri]

Hi folks,

> If you use an unstable MV, you probably won't be able to upgrade your 
> software. Because whenever something changes, you'll probably get 
> serialization exceptions being thrown inside the controller. Fatal ones.

Thanks for this clarification. I think this concrete risk should be
highlighted in the KIP and in the "unstable.metadata.versions.enable"
documentation.

In the test plan, should we also have one system test checking that
"features with a stable MV will never have that MV changed"?

On Wed, Jan 10, 2024 at 8:16 AM Colin McCabe  wrote:
>
> On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote:
> > Hi folks,
> >
> > Thank you for the questions.
> >
> > Let me clarify about reorder first. The reorder of unstable metadata
> > versions should be infrequent.
>
> Why does it need to be infrequent? We should be able to reorder unstable 
> metadata versions as often as we like. There are no guarantees about unstable 
> MVs.
>
> > The time you reorder is when a feature that
> > requires a higher metadata version to enable becomes "production ready" and
> > the features with unstable metadata versions less than the new stable one
> > are moved to metadata versions greater than the new stable feature. When we
> > reorder, we are always allocating a new MV and we are never reusing an
> > existing MV even if it was also unstable. This way a developer upgrading
> > their environment with a specific unstable MV might see existing
> > functionality stop working but they won't see new MV dependent
> > functionality magically appear. The feature set for a given unstable MV
> > version can only decrease with reordering.
>
> If you use an unstable MV, you probably won't be able to upgrade your 
> software. Because whenever something changes, you'll probably get 
> serialization exceptions being thrown inside the controller. Fatal ones.
>
> Given that this is true, there's no reason to have special rules about what 
> we can and can't do with unstable MVs. We can do anything.
>
> >
> > How do we define "production ready" and when should we bump
> > LATEST_PRODUCTION? I would like to define it to be the point where the
> > feature is code complete with tests and the KIP for it is approved. However
> > even with this definition if the feature later develops a major issue it
> > could still block future features until the issue is fixed which is what we
> > are trying to avoid here. We could be much more formal about this and let
> > the release manager for a release define what is stable for a given release
> > and then do the bump just after the branch is created on the branch. When
> > an RC candidate is accepted, the bump would be backported. I would like to
> > hear other ideas here.
> >
>
> Yeah, it's an interesting question. Overall, I think developers should define 
> when a feature is production ready.
>
> The question to ask is, "are you ready to take this feature to production in 
> your workplace?" I think most developers do have a sense of this. Obviously 
> bugs and mistakes can happen, but I think this standard would avoid most of 
> the issues that we're trying to avoid by having unstable MVs in the first 
> place.
>
> ELR is a good example. Nobody would have said that it was production ready in 
> 3.7 ... hence it belonged (and still belongs) in an unstable MV, until that 
> changes (hopefully soon :) )
>
> best,
> Colin
>
> > --Proven
> >
> > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe  wrote:
> >
> >> Hi Justine,
> >>
> >> Yes, this is an important point to clarify. Proven can comment more, but
> >> my understanding is that we can do anything to unstable metadata versions.
> >> Reorder them, delete them, change them in any other way. There are no
> >> stability guarantees. If the current text is unclear let's add more
> >> examples of what we can do (which is anything) :)
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote:
> >> > Hey Colin,
> >> >
> >> > I had some offline discussions with Proven previously and it seems like
> >> he
> >> > said something different so I'm glad I brought it up here.
> >> >
> >> > Let's clarify if we are ok with reordering unstable metadata versions :)
> >> >
> >> > Justine
> >> >
> >> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe  wrote:
> >> >
> >> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
> >> >> > Hey all,
> >> >> >
> >> >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata
> >> >> > version. Is this something we should do as soon as the feature is
> >> >> complete
> >> >> > or something we do when we are releasing kafka. When is the time we
> >> >> abandon
> >> >> > a MV so that other features can be unblocked?
> >> >>
> >> >> Hi Justine,
> >> >>
> >> >> Thanks for reviewing.
> >> >>
> >> >> The idea is that you should bump LATEST_PRODUCTION when you want to
> >> take a
> >> >> feature to production. That could mean deploying it internally
> >> somewhere to
> >> >> 

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Federico Valeri]

On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits
 wrote:
>
> Hi Proven,
>
> I'd say that we should do 2 & 2.  The idea is that for small features that
> can be done and stabilized within a short period of time (with one or very
> few commits) that's exactly what happens -- people interested in testing
> in-progress feature could take unstable code from a patch (or private
> branch / fork) with the expectation that that private code could create a
> state that will not be compatible with anything (or may be completely
> broken for that matter -- in the end of the day it's a functionality that
> may not be fully tested or even fully implemented); and once the feature is
> stable it goes to trunk it is fully committed there, if the bugs are found
> they'd get fixed "forward".

I agree with this reasoning.

> The 2 & 2 option pretty much extends this to
> large features -- if a feature is above stable MV, then going above it is
> like getting some in-progress code for early testing with the expectation
> that something may not fully work or leave system in upgradable state;

Usually I expect that an early access feature may not fully work, but
not that it could affect upgrades. I think this is less obvious,
that's why I asked to document clearly.

> promoting a feature into a state MV would come with the expectation that
> the feature gets fully committed and any bugs will be fixed "forward".
>
> -Artem
>
> On Thu, Jan 11, 2024 at 10:16 AM Proven Provenzano
>  wrote:
>
> > We have two approaches here for how we update unstable metadata versions.
> >
> >1. The update will only increase MVs of unstable features to a value
> >greater than the new stable feature. The idea is that a specific
> > unstable
> >MV may support some set of features and in the future that set is
> > always a
> >strict subset of the current set. The issue is that moving a feature to
> >make way for a stable feature with a higher MV will leave holes.
> >2. We are free to reorder the MV for any unstable feature. This removes
> >the hole issue, but does make the unstable MVs more muddled. There isn't
> >the same binary state for a MV where a feature is available or there is
> > a
> >hole.
> >
> >
> > We also have two ends of the spectrum as to when we update the stable MV.
> >
> >1. We update at release points which reduces the amount of churn of the
> >unstable MVs and makes a stronger correlation between accepted features
> > and
> >stable MVs for a release but means less testing on trunk as a stable MV.
> >2. We update when the developers of a feature think it is done. This
> >leads to features being available for more testing in trunk but forces
> > the
> >next release to include it as stable.
> >
> >
> > I'd like more feedback from others on these two dimensions.
> > --Proven
> >
> >
> >
> > On Wed, Jan 10, 2024 at 12:16 PM Justine Olshan
> >  wrote:
> >
> > > Hmm it seems like Colin and Proven are disagreeing with whether we can
> > swap
> > > unstable metadata versions.
> > >
> > > >  When we reorder, we are always allocating a new MV and we are never
> > > reusing an existing MV even if it was also unstable.
> > >
> > > > Given that this is true, there's no reason to have special rules about
> > > what we can and can't do with unstable MVs. We can do anything
> > >
> > > I don't have a strong preference either way, but I think we should agree
> > on
> > > one approach.
> > > The benefit of reordering and reusing is that we can release features
> > that
> > > are ready earlier and we have more flexibility. With the approach where
> > we
> > > always create a new MV, I am concerned with having many "empty" MVs. This
> > > would encourage waiting until the release before we decide an incomplete
> > > feature is not ready and moving its MV into the future. (The
> > > abandoning comment I made earlier -- that is consistent with Proven's
> > > approach)
> > >
> > > I think the only potential issue with reordering is that it could be a
> > bit
> > > confusing and *potentially *prone to errors. Note I say potentially
> > because
> > > I think it depends on folks' understanding with this new unstable
> > metadata
> > > version concept. I echo Federico's comments about making sure the risks
> > are
> > > highlighted.
> > >
> > > Thanks,
> > >
> > > Just

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

[Federico Valeri]

+1 non binding

Thanks

On Fri, Jan 12, 2024 at 1:31 AM Boudjelda Mohamed Said
 wrote:
>
> +1 (binding)
>
>
> On Fri, Jan 12, 2024 at 1:21 AM Satish Duggana 
> wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Satish.
> >
> > On Thu, 11 Jan 2024 at 17:52, Divij Vaidya 
> > wrote:
> > >
> > > +1 (binding)
> > >
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Tue, Dec 26, 2023 at 7:05 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > --
> > > > Kamal
> > > >
> > > > On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov 
> > > > wrote:
> > > >
> > > > > Heya all!
> > > > >
> > > > > KIP-1005 (
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset
> > > > > )
> > > > > has been open for around a month with no further comments - I would
> > like
> > > > to
> > > > > start a voting round on it!
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > >
> >

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Federico Valeri]

On Fri, Dec 29, 2023 at 8:49 AM ziming deng  wrote:
>
> Hello all,
>
> We have 2 people prefer using "--enable-incremental" and it makes sense to 
> move the incompatible process to 4.X, I changed the default way to still use 
> `alterConfigs` and use `incrementalAlterConfigs` only when 
> "--enable-incremental" is specified.
>

I agree with this change, much better. Thanks.

> I will initiate a vote process if no further opinions coming.
>
> Best,
> Ziming
>
>
> > On Dec 26, 2023, at 13:21, Kamal Chandraprakash 
> >  wrote:
> >
> > Hi Ziming,
> >
> > Thanks for the KIP! The proposal LGTM.
> >
> > I'm also inclined towards option 2 (i.e. add an explicit
> > --enable-incremental flag in 3.X version)
> > to avoid any incompatible change in v3.X. As mentioned in this thread, many
> > users might be
> > using external tools to do the topic rebalance and apply throttle config on
> > the brokers, it might
> > break for them when the tools are upgraded before the broker binary.
> >
> > cruise-control is currently using kafka-version: 3.1
> > https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_5/gradle.properties#L5
> >
> > On Mon, Dec 25, 2023 at 7:15 PM Divij Vaidya 
> > wrote:
> >
> >> Thank you for the summary, Ziming.
> >>
> >> Personally, I would prefer the latter i.e. having the incompatible change
> >> in 4.x instead of 3.x. This is because a major version upgrade goes through
> >> additional scrutiny by the users and usually comes with inevitable code
> >> changes required on the client. Hence, this incompatibility will be part of
> >> one amongst many changes that users will perform to upgrade to 4.x. This is
> >> unlike a major version change from 3.7 to 3.8 where users expect a simple
> >> upgrade without any code changes.
> >>
> >> Let's wait and hear what others think about this.
> >>
> >> --
> >> Divij Vaidya
> >>
> >>
> >>
> >> On Mon, Dec 25, 2023 at 1:18 PM ziming deng 
> >> wrote:
> >>
> >>> Hello Divij Vaidya,
> >>>
> >>> You are right that users should update existing scripts to add
> >>> ‘—disable-incremental’, and you mentioned another upgrade path which is
> >>> similar, the summary of the 2 schemes are:
> >>> we change existing scripts to use `incrementalAlterConfigs` and add
> >>> "--disable-incremental" flag for old servers in Kafka 3.X, and remove it
> >> in
> >>> Kafka 4.X.
> >>> we keep existing scripts unchanged and add an "--enable-incremental" flag
> >>> for new servers in Kafka 3.X, and remove it in Kafka 4.X.
> >>>
> >>> I think there will always be an incompatible upgrade process to move
> >>> default behavior from `alterConfigs` to `incrementalConfigs`. In the
> >> first
> >>> scheme we are doing this incompatible upgrade in Kafka 3.X, and in the
> >>> second scheme we are moving it to 4.X, I think we should do it as soon as
> >>> possible if it's inevitable.
> >>> However, I will add this to , and I'm open to this
> >>> if more people think it's more suitable.
> >>>
> >>>
> >>> --,
> >>> Ziming
> >>>
>  On Dec 22, 2023, at 18:13, Divij Vaidya 
> >> wrote:
> 
>  Divij Vaidya
> >>>
> >>>
> >>
>

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

[Federico Valeri]

Thanks! This will finally reconcile Javadoc and implementation.

+1 (non binding)

On Thu, Jan 4, 2024 at 6:49 AM Yash Mayya  wrote:
>
> Hi Chris,
>
> +1 (binding), thanks for the KIP.
>
> Based on discussion in other threads, it looks like the community is
> aligned with having a 3.8 release before the 4.0 release so we should be
> able to remove the 'tasks.max.enforce' connector property in 4.0 (we'd
> discussed potentially having to live with this property until 5.0 in this
> KIP's discussion thread). Once we have confirmation of a 3.8 release, will
> this KIP be updated to reflect the exact AK versions where the deprecated
> property will be introduced and removed?
>
> Thanks,
> Yash
>
> On Wed, Jan 3, 2024 at 11:37 PM Greg Harris 
> wrote:
>
> > Hey Chris,
> >
> > Thanks for the KIP! I think the aggressive default and deprecation
> > schedule is the right choice for this change.
> >
> > +1 (binding)
> >
> > On Wed, Jan 3, 2024 at 9:01 AM Mickael Maison 
> > wrote:
> > >
> > > Hi Chris,
> > >
> > > +1 (binding), thanks for the KIP
> > >
> > > Mickael
> > >
> > > On Tue, Jan 2, 2024 at 8:55 PM Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > >  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks Chris!
> > > >
> > > > From: dev@kafka.apache.org At: 01/02/24 11:49:18 UTC-5:00To:
> > dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka
> > Connect
> > > >
> > > > Hi all,
> > > >
> > > > Happy New Year! Wanted to give this a bump now that the holidays are
> > over
> > > > for a lot of us. Looking forward to people's thoughts!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Dec 4, 2023 at 10:36 AM Chris Egerton  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-1004, which adds enforcement for
> > the
> > > > > tasks.max connector property in Kafka Connect.
> > > > >
> > > > > The KIP:
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+
> > > > property+in+Kafka+Connect
> > > > >
> > > > > The discussion thread:
> > > > > https://lists.apache.org/thread/scx75cjwm19jyt19wxky41q9smf5nx6d
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > > >
> >

Re: [VOTE] 3.5.2 RC1

[Federico Valeri]

Hi Luke,

- Compiled from source (Java 17 and Scala 2.13)
- Ran unit and integration tests
- Ran custom client apps using staging artifacts

+1 (non binding)

Thanks
Fede

On Wed, Nov 22, 2023 at 2:44 PM Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks for running the release.
> I did the following:
> - Verified artifact's signatures and hashes
> - Checked JavaDoc (with navigation to Oracle JavaDoc)
> - Compiled source code
> - Run unit tests and integration tests
> - Run getting started with ZK and KRaft
>
> It gets a +1 from my side (non-binding)
>
> Best,
>
> On Tue, Nov 21, 2023 at 11:09 AM Luke Chen  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.5.2.
> >
> > This is a bugfix release with several fixes since the release of 3.5.1,
> > including dependency version bumps for CVEs.
> >
> > Release notes for the 3.5.2 release:
> > https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Nov. 28.
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~showuon/kafka-3.5.2-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~showuon/kafka-3.5.2-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.5 branch) is the 3.5.2 tag:
> > https://github.com/apache/kafka/releases/tag/3.5.2-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/35/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/35/protocol.html
> >
> > * Successful Jenkins builds for the 3.5 branch:
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/98/
> > There are some falky tests, including the testSingleIP test failure. It
> > failed because of some infra change and we fixed it
> >  recently.
> >
> > System tests: running, will update the results later.
> >
> >
> >
> > Thank you.
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] KIP-974: Docker Image for GraalVM based Native Kafka Broker

[Federico Valeri]

Hi,

+1 (non binding)

Thanks

On Wed, Nov 22, 2023 at 3:16 PM Manikumar  wrote:
>
> Hi Krishna,
>
> Thanks for KIP.  +1 (binding).
>
>
> Thanks,
> Manikumar
>
> On Mon, Nov 20, 2023 at 11:57 AM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi,
> > I'd like to call a vote on KIP-974 which aims to publish a docker image for
> > GraalVM based Native Kafka Broker.
> >
> > KIP -
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> >
> > Discussion thread -
> > https://lists.apache.org/thread/98wnx4w92fqj5wymkqlqyjsvzxz277hk
> >
> > Regards,
> > Krishna
> >

Re: [VOTE] 3.6.1 RC0

[Federico Valeri]

Hi Mickael,

- Build from source (Java 17, Scala 2.13)
- Run unit and integration tests
- Run custom client apps using staging artifacts

+1 (non binding)

Thanks
Fede



On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz  wrote:
>
> +1 non-binding. I used the staged Scala 2.13 artifacts and the staged Maven
> repo for my tests. All seems to work fine.
>
> Thanks
> Jakub
>
> On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.6.1.
> >
> > This is a bugfix release with several fixes, including dependency
> > version bumps for CVEs.
> >
> > Release notes for the 3.6.1 release:
> > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday, December 1
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
> >
> > PR for updating docs:
> > https://github.com/apache/kafka-site/pull/568
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > Unit/integration tests: We still have a lot of flaky tests in the 3.6
> > branch. Looking at the last few 3.6 builds in
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it seems all
> > tests passed at least once apart from
> > ClusterConnectionStatesTest.testSingleIP(). There's
> > https://issues.apache.org/jira/browse/KAFKA-15762 to fix that test.
> > System tests: Still running I'll post an update once they complete.
> >
> > Thanks,
> > Mickael
> >

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Federico Valeri]

+1 (non binding)

Thanks.

On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
>
> Hi Ziming,
>
> +1(binding) from me.
>
> Thanks.
> Luke
>
> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > +1 (non-binding)
> >
> > On Mon, Jan 22, 2024 at 8:34 AM ziming deng 
> > wrote:
> >
> >> Hello everyone,
> >> I'd like to initiate a vote for KIP-1011.
> >> This KIP is about replacing alterConfigs with incrementalAlterConfigs
> >> when updating broker configs using kafka-configs.sh, this is similar to
> >> what we have done in KIP-894.
> >>
> >> KIP link:
> >> KIP-1011: Use incrementalAlterConfigs when updating broker configs by
> >> kafka-configs.sh - Apache Kafka - Apache Software Foundation
> >> 
> >> cwiki.apache.org
> >> 
> >> [image: favicon.ico]
> >> 
> >> 
> >>
> >> Discussion thread:
> >>
> >>
> >> lists.apache.org
> >> 
> >> 
> >> 
> >>
> >>
> >> --,
> >> Best,
> >> Ziming
> >>
> >

[REVIEW REQUEST] Move StorageTool to tools

[Federico Valeri]

Hi folks, this PR has been open for some time now, can anyone have a look?

https://github.com/apache/kafka/pull/14847

Thanks.

Re: the migration of command tools

[Federico Valeri]

Hi, if a tool already has a wrapper bash script in bin and
bin/windows, then there is no need to create a redirection in Scala,
as users are supposed to use the script. If there is any ST left which
is using the tool directly, a bash script should be created, and the
ST changed to use the script with new releases only.

On Wed, Apr 10, 2024 at 3:08 PM Chia-Ping Tsai  wrote:
>
> hi David
>
> thanks for quickly response!!
>
> According to KIP-906, the BC rules are
>
> 1) The old package name must be deprecated in the target release (e.g. 3.5)
> and redirection removed in the next major release (e.g. 4.0).
> 2) Existing users will get a deprecation warning when using the old package
> name, while old SPIs and classes will be marked as deprecated.
>
> I will file a jira to make sure we don't violate that rules
>
> Best,
> Chia-Ping
>
> David Jacot  於 2024年4月10日 週三 下午8:57寫道：
>
> > Hey,
> >
> > I think that we discussed this in this KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > .
> > I don't remember all the details though.
> >
> > Best,
> > David
> >
> > On Wed, Apr 10, 2024 at 2:54 PM Chia-Ping Tsai  wrote:
> >
> > > Dear Kafka,
> > >
> > > Migrating command tools from core module to tools module is not news.
> > > However, I want to make sure I don't misunderstand the BC rules.
> > >
> > > The question is "Should we keep origin class?"
> > >
> > > FeatureCommand (
> > >
> > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/FeatureCommand.scala
> > > )
> > > is a good example. We keep the origin class file due to backward
> > > compatibility. However, we don't do that to other tools.
> > >
> > > It seems to me that we should align the BC rules for all tools. And here
> > is
> > > my two cents: the expected way of using command tool is by script file,
> > so
> > > we DON'T need to keep origin class file.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Chia-Ping
> > >
> >

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

[Federico Valeri]

Hi Kamal, it looks like all TS configurations starts with "remote."
prefix, so I was wondering if we should name it
"remote.fetch.max.wait.ms".

On Fri, Apr 26, 2024 at 7:07 PM Kamal Chandraprakash
 wrote:
>
> Hi all,
>
> If there are no more comments, I'll start a vote thread by tomorrow.
> Please review the KIP.
>
> Thanks,
> Kamal
>
> On Sat, Mar 30, 2024 at 11:08 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > Bumping the thread. Please review this KIP. Thanks!
> >
> > On Thu, Feb 1, 2024 at 9:11 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> >> Hi Jorge,
> >>
> >> Thanks for the review! Added your suggestions to the KIP. PTAL.
> >>
> >> The `fetch.max.wait.ms` config will be also applicable for topics
> >> enabled with remote storage.
> >> Updated the description to:
> >>
> >> ```
> >> The maximum amount of time the server will block before answering the
> >> fetch request
> >> when it is reading near to the tail of the partition (high-watermark) and
> >> there isn't
> >> sufficient data to immediately satisfy the requirement given by
> >> fetch.min.bytes.
> >> ```
> >>
> >> --
> >> Kamal
> >>
> >> On Thu, Feb 1, 2024 at 12:12 AM Jorge Esteban Quilcate Otoya <
> >> quilcate.jo...@gmail.com> wrote:
> >>
> >>> Hi Kamal,
> >>>
> >>> Thanks for this KIP! It should help to solve one of the main issues with
> >>> tiered storage at the moment that is dealing with individual consumer
> >>> configurations to avoid flooding logs with interrupted exceptions.
> >>>
> >>> One of the topics discussed in [1][2] was on the semantics of `
> >>> fetch.max.wait.ms` and how it's affected by remote storage. Should we
> >>> consider within this KIP the update of `fetch.max.wail.ms` docs to
> >>> clarify
> >>> it only applies to local storage?
> >>>
> >>> Otherwise, LGTM -- looking forward to see this KIP adopted.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/KAFKA-15776
> >>> [2] https://github.com/apache/kafka/pull/14778#issuecomment-1820588080
> >>>
> >>> On Tue, 30 Jan 2024 at 01:01, Kamal Chandraprakash <
> >>> kamal.chandraprak...@gmail.com> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > I have opened a KIP-1018
> >>> > <
> >>> >
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests
> >>> > >
> >>> > to introduce dynamic max-remote-fetch-timeout broker config to give
> >>> more
> >>> > control to the operator.
> >>> >
> >>> >
> >>> >
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests
> >>> >
> >>> > Let me know if you have any feedback or suggestions.
> >>> >
> >>> > --
> >>> > Kamal
> >>> >
> >>>
> >>

Re: [VOTE] KIP-909: DNS Resolution Fallures Should Not Fail the Client

[Federico Valeri]

+1 (non binding)
Thanks

On Mon, Apr 29, 2024 at 10:18 AM Rajini Sivaram  wrote:
>
> Hi Philip,
>
> +1 (binding). Thanks for the KIP!
>
> Regards,
>
> Rajini
>
>
> On Tue, Apr 25, 2023 at 8:52 PM Philip Nee  wrote:
>
> > Thanks for the vote.  We've decided to make a minor change to the default
> > timeout from 5min to 2min.
> >
> > On Tue, Apr 25, 2023 at 11:42 AM David Jacot 
> > wrote:
> >
> > > +1 (binding) Thanks for the KIP, Philip!
> > >
> > > Le mar. 25 avr. 2023 à 20:23, José Armando García Sancio
> > >  a écrit :
> > >
> > > > +1. Thanks for the design. Looking forward to the implementation.
> > > >
> > > > On Tue, Apr 25, 2023 at 10:49 AM Jason Gustafson
> > > >  wrote:
> > > > >
> > > > > +1 Thanks Philip!
> > > > >
> > > > >
> > > > > On Thu, Apr 13, 2023 at 7:49 AM Kirk True  wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > > On Apr 10, 2023, at 1:53 PM, Philip Nee 
> > > wrote:
> > > > > > >
> > > > > > > Hey everyone!
> > > > > > >
> > > > > > > I'm starting a vote for KIP-909: DNS Resolution Fallures Should
> > Not
> > > > Fail
> > > > > > > the Client  > > > Clients>
> > > > > > >
> > > > > > > Please refer to the discussion thread here:
> > > > > > > https://lists.apache.org/thread/st84zzwnq5m3pkzd1r7jk9lmqdt9m98s
> > > > > > >
> > > > > > > Thanks!
> > > > > > > P
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -José
> > > >
> > >
> >

Re: [VOTE] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

[Federico Valeri]

+1 non binding

Thanks

On Wed, May 8, 2024 at 5:27 PM Andrew Schofield
 wrote:
>
> Hi,
> Thanks for the KIP.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 8 May 2024, at 15:48, David Jacot  wrote:
> >
> > Hi folks,
> >
> > I'd like to start a voting thread for KIP-1041: Drop
> > `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8).
> >
> > KIP: https://cwiki.apache.org/confluence/x/9YobEg
> >
> > Best,
> > David
>

Re: [VOTE] KIP-1018: Introduce max remote fetch timeout config

[Federico Valeri]

+1 non binding

Thanks

On Thu, May 9, 2024 at 12:05 PM Luke Chen  wrote:
>
> Hi Kamal,
>
> Thanks for the KIP!
> +1 from me.
>
> Thanks.
> Luke
>
> On Mon, May 6, 2024 at 5:03 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > We would like to start a voting thread for KIP-1018: Introduce
> > max remote fetch timeout config for DelayedRemoteFetch requests.
> >
> > The KIP is available on
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests
> >
> > If you have any suggestions, feel free to participate in the discussion
> > thread:
> > https://lists.apache.org/thread/9x21hzpxzmrt7xo4vozl17d70fkg3chk
> >
> > --
> > Kamal
> >

Re: [DISCUSS] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

[Federico Valeri]

Hi David, I can't think about any valid use case where changing the
default value would be useful, and your motivation is convincing.
Thanks.

On Fri, May 3, 2024 at 6:01 AM Andrew Schofield
 wrote:
>
> Hi David,
> I think this KIP is a very good idea. It would be good to get rid of this 
> cruft.
>
> Thanks,
> Andrew
>
> > On 2 May 2024, at 18:54, David Jacot  wrote:
> >
> > Hi folks,
> >
> > I have put together a very small KIP to
> > deprecate offsets.commit.required.acks in 3.8 and remove it in 4.0. See the
> > motivation for the reason.
> >
> > KIP: https://cwiki.apache.org/confluence/x/9YobEg
> >
> > Please let me know what you think.
> >
> > Best,
> > David
>

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

[Federico Valeri]

Hi Fred, this is a useful addition.

+1 non binding

Thanks

On Fri, May 3, 2024 at 4:11 PM Andrew Schofield
 wrote:
>
> Hi Fred,
> Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a 
> worthwhile improvement.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 30 Apr 2024, at 14:02, Frédérik Rouleau  
> > wrote:
> >
> > Hi all,
> >
> > As there is no more activity for a while on the discuss thread, I think we
> > can start a vote.
> > The KIP is available on
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1036%3A+Extend+RecordDeserializationException+exception
> >
> >
> > If you have some feedback or suggestions, please participate to the
> > discussion thread:
> > https://lists.apache.org/thread/or85okygtfywvnsfd37kwykkq5jq7fy5
> >
> > Best regards,
> > Fred
>

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

[Federico Valeri]

Hello Alieh, thanks for this useful KIP.

There is a typo in the motivation when you talk about the
UnknownTopicOrPartitionException. It's delivery.timeout.ms, not
deliver.timeout.ms.

In the past, I did some work to improve and clean the official Kafka
examples, which I think are useful for new Kafka users. I was
wondering if it is worth to improve them in order to show the correct
usage of this new interface. If you agree, maybe we could mention this
in the proposed changes.

> The accepted responses for RecordTooLargeException are FAIL and SWALLOW. 
> Therefore, RETRY will be interpreted and executed as FAIL.

Why do we need this javadoc note? I think it's not possible to return
RETRY in the current form.

When we talk about swallowing in the default implementation, I think
we will log an error/warning and drop the record right? If yes, should
we clarify this and improve the DROP_INVALID_LARGE_RECORDS_DOC by
mentioning the logging part?

Should we mention somewhere which logic takes precedence when both the
interface and configs are used?

On Tue, May 14, 2024 at 4:45 PM Chris Egerton  wrote:
>
> Hi Alieh,
>
> Thank you for all the updates! One final question--how will the retry
> timeout for unknown topic partition errors be implemented? I think it would
> be best if this could be done with an implementation of the error handler,
> but I don't see a way to track the necessary information with the
> current ProducerExceptionHandler interface.
>
> Cheers,
>
> Chris
>
> On Tue, May 14, 2024 at 9:10 AM Alieh Saeedi 
> wrote:
>
> > Thanks Andrew. Done :)
> >
> > @Chris: I changed the config parameter type from boolean to integer, which
> > defines the timeout for retrying. I thought reusing `max.block.ms` was not
> > reasonable as you mentioned.
> >
> > So if the KIP looks good, let 's skip to the good part ;-) VOTING :)
> >
> > Bests,
> > Alieh
> >
> >
> >
> >
> >
> > On Tue, May 14, 2024 at 12:26 PM Andrew Schofield <
> > andrew_schofi...@live.com>
> > wrote:
> >
> > > Hi Alieh,
> > > Just one final comment.
> > >
> > > [AJS5] Existing classes use Retriable, not Retryable. For example:
> > >
> > >
> > https://kafka.apache.org/21/javadoc/org/apache/kafka/common/errors/RetriableException.html
> > >
> > > I suggest RetriableResponse and NonRetriableResponse.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 13 May 2024, at 23:17, Alieh Saeedi 
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > Thanks for all the valid points you listed.
> > > >
> > > >
> > > > KIP updates and addressing concerns:
> > > >
> > > >
> > > > 1) The KIP now suggests two Response types: `RetryableResponse` and
> > > > `NonRetryableResponse`
> > > >
> > > >
> > > > 2) `custom.exception.handler` is changed to
> > > `custom.exception.handler.class`
> > > >
> > > >
> > > > 3) The KIP clarifies that `In the case of an implemented handler for
> > the
> > > > specified exception, the handler takes precedence.`
> > > >
> > > >
> > > > 4)  There is now a `default` implementation for both handle() methods.
> > > >
> > > >
> > > > 5)  @Chris: for `UnknownTopicOrPartition`, the default is already
> > > retrying
> > > > for 60s. (In fact, the default value of `max.block.ms`). If the
> > handler
> > > > instructs to FAIL or SWALLOW, there will be no retry, and if the
> > handler
> > > > instructs to RETRY, that will be the default behavior, which follows
> > the
> > > > values in already existing config parameters such as `max.block.ms`.
> > > Does
> > > > that make sense?
> > > >
> > > >
> > > > Hope the changes and explanations are convincing :)
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Alieh
> > > >
> > > > On Mon, May 13, 2024 at 6:40 PM Justine Olshan
> > > 
> > > > wrote:
> > > >
> > > >> Oh I see. The type isn't the error type but a newly defined type for
> > the
> > > >> response. Makes sense and works for me.
> > > >>
> > > >> Justine
> > > >>
> > > >> On Mon, May 13, 2024 at 9:13 AM Chris Egerton <
> > fearthecel...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> If we have dedicated methods for each kind of exception
> > > >>> (handleRecordTooLarge, handleUnknownTopicOrPartition, etc.), doesn't
> > > that
> > > >>> provide sufficient constraint? I'm not suggesting we eliminate these
> > > >>> methods, just that we change their return types to something more
> > > >> flexible.
> > > >>>
> > > >>> On Mon, May 13, 2024, 12:07 Justine Olshan
> > >  > > >>>
> > > >>> wrote:
> > > >>>
> > >  I'm not sure I agree with the Retriable and NonRetriableResponse
> > > >> comment.
> > >  This doesn't limit the blast radius or enforce certain errors are
> > > used.
> > >  I think we might disagree on how controlled these interfaces can
> > be...
> > > 
> > >  Justine
> > > 
> > >  On Mon, May 13, 2024 at 8:40 AM Chris Egerton
> >  > > >>>
> > >  wrote:
> > > 
> > > > Hi Alieh,
> > > >
> > > > Thanks for the updates! I just have a few more thoughts:
> > > >
> > > > - I don't think 

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Federico Valeri]

Hi Philip, thanks for this KIP. As other mentioned, I think it is
useful in dynamic environment like Kubernetes.

I was wondering if we should also update the "examples" modules with
this new non-retriable exception. Wdyt?

On Fri, Apr 21, 2023 at 11:13 PM Philip Nee  wrote:
>
> Hey Jason,
>
> Thanks again. I've updated the KIP.
>
> P
>
> On Fri, Apr 21, 2023 at 9:56 AM Jason Gustafson 
> wrote:
>
> > Hey Philip, that sounds good to me.
> >
> > -Jason
> >
> >
> >
> > On Thu, Apr 20, 2023 at 1:20 PM Philip Nee  wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for the response. I agree with your suggestions. Just to clarify,
> > we
> > > want the timeout, bootstrap.resolve.timeout.ms to only bound the DNS
> > > resolution time, and we can throw a fatal, BootstrapResolutionException
> > (so
> > > not connection exception anymore) afterward.
> > >
> > > I think that aligns with the goal of this KIP.
> > >
> > > P
> > >
> > > On Thu, Apr 20, 2023 at 9:23 AM Jason Gustafson
> >  > > >
> > > wrote:
> > >
> > > > Hey Philip,
> > > >
> > > > Yeah, I see your point. Here is the challenge I'm considering. Today,
> > the
> > > > client does not forward connection-related errors to the application.
> > It
> > > is
> > > > debatable whether it should, but that is the behavior that applications
> > > > expect today. The only case of a fatal error is DNS resolution and the
> > > > application must handle it because they cannot even construct the
> > client.
> > > > Once we have this KIP, we will introduce a separate fatal failure
> > > mechanism
> > > > through the `BootstrapConnectionException`. And we will use it not only
> > > for
> > > > DNS failures, but general bootstrap connection failures . The problem
> > is
> > > > that existing applications will not know that this should be treated
> > as a
> > > > fatal error. So they may continue to retry. Does it help in this
> > > situation
> > > > to poison the client so that it cannot be used? I'm not sure. Perhaps
> > it
> > > > would reduce the risk a bit if we change `
> > > bootstrap.connection.timeout.ms`
> > > > to `bootstrap.resolve.timeout.ms` or something like that? Then we
> > retain
> > > > the current behavior if DNS succeeds, but connections fail. I know I'm
> > > the
> > > > one that suggested generalizing the configuration, but I feel some
> > > > hesitation after thinking about it more.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Wed, Apr 19, 2023 at 8:10 PM Philip Nee 
> > wrote:
> > > >
> > > > > Hey Jason,
> > > > >
> > > > > Thanks for your review.  I think if we make it a retriable error,
> > does
> > > it
> > > > > make sense to have a configurable timeout still? as we expect the
> > user
> > > to
> > > > > continue to retry anyway.
> > > > >
> > > > > I'm considering the case of bad configuration. If the user retries
> > the
> > > > > error, then we rely on the error/warning to alert the user.  In this
> > > > case,
> > > > > maybe we continue using the proposed behavior, i.e. warn on each poll
> > > > after
> > > > > the timeout period.
> > > > >
> > > > > If you agree that a configuration is needed, maybe we can call this
> > > > > *bootstrap.auto.retry.ms
> > > > >  *instead, to indicate a
> > configurable
> > > > > period of automatic retry. What do you think?
> > > > >
> > > > > Cheers,
> > > > > P
> > > > >
> > > > > On Wed, Apr 19, 2023 at 7:17 PM Jason Gustafson
> > > >  > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Phillip,
> > > > > >
> > > > > > The KIP looks good. 5 minutes seems like a reasonable tradeoff. I
> > do
> > > > > wonder
> > > > > > if it is necessary to treat bootstrap timeout as a fatal error
> > > though.
> > > > It
> > > > > > seems possible that the exception might be caught by handlers in
> > > > existing
> > > > > > applications which may not expect that the client needs to be
> > > > restarted.
> > > > > > Perhaps it would be safer to make it retriable? As long as the
> > > > > application
> > > > > > continues trying to use the client, we could continue trying to
> > reach
> > > > the
> > > > > > bootstrap servers perhaps? That would be closer to behavior today
> > > which
> > > > > > only treats DNS resolution failures as fatal. What do you think?
> > > > > >
> > > > > > Best,
> > > > > > Jason
> > > > > >
> > > > > > On Mon, Apr 10, 2023 at 1:53 PM Philip Nee 
> > > > wrote:
> > > > > >
> > > > > > > Thanks, everyone: I'm starting a vote today.  Here's the recap
> > for
> > > > some
> > > > > > of
> > > > > > > the questions:
> > > > > > >
> > > > > > > John: I changed the proposal to throw a non-retriable exception
> > > after
> > > > > the
> > > > > > > timeout elapses. I feel it might be necessary to poison the
> > client
> > > > > after
> > > > > > > retry expires, as it might indicate a real issue.
> > > > > > > Ismael: The proposal is to add a configuration for the retry and
> > it
> > > > > will
> > > > > > > throw a non-retriable exception after the time 

Re: [ANNOUNCE] New committer: Christo Lolov

[Federico Valeri]

Congrats!

On Tue, Mar 26, 2024 at 2:27 PM Mickael Maison  wrote:
>
> Congratulations Christo!
>
> On Tue, Mar 26, 2024 at 2:26 PM Chia-Ping Tsai  wrote:
> >
> > Congrats Christo!
> >
> > Chia-Ping

[DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

[Federico Valeri]

Hi all,

I'd like to kick off a discussion for KIP-1057, that proposes to add
remote log metadata flag to the dump log tool, which is useful when
debugging.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool

Thanks,
Fede

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

[Federico Valeri]

Hi Divij,

On Sun, Jun 16, 2024 at 7:38 PM Divij Vaidya  wrote:
>
> Hello Federico
>
> Please note that the topic-based RLMM is one of the possible
> implementations of RLMM. Hence, whatever solution we design here should: 1\
> be explicit that this tooling only works for topic based RLMM 2\ specify
> the handling of the failure mode when topic based RLMM is not being used.
>

That's true, thanks for pointing out.

> I would argue that Topic based RLMM cannot be treated the same as other
> internal topics. Topic based RLMM topic is an optional topic which can have
> any possible schema (depending on plugin implementation) whereas
> other internal topics are always guaranteed to be present with a fixed
> schema.
>

Right, I updated the KIP with an improved option description.

> In light of the above statements, the rejected alternative sounds better to
> me because:
> 1\ it provides the ability to dump logs for "any" RLMM implementation and
> not just topic based RLMM.
> 2\ we don't have to deal with schema evolution of topic based RLMM in this
> tool. That responsibility will be delegated to the decoder class which the
> operator can define using the flag "--value-decoder-class".
>
> Is there a reason that you are unable to use the rejected solution (which
> requires no changes) for debugging purposes?
>

The rejected alternative will still be available, but I thought that
having a dedicated flag would make debugging easier, as I guess most
people will use the default RLMM implementation. I would be happy to
hear other opinions on this.

> --
> Divij Vaidya
>
>
>
> On Sat, Jun 15, 2024 at 4:43 PM Federico Valeri 
> wrote:
>
> > Hi all,
> >
> > I'd like to kick off a discussion for KIP-1057, that proposes to add
> > remote log metadata flag to the dump log tool, which is useful when
> > debugging.
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
> >
> > Thanks,
> > Fede
> >