Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Colin McCabe]

On Tue, Jan 9, 2024, at 17:07, Jason Gustafson wrote:
> Hi Jose,
>
> Thanks for the KIP! A few initial questions below:
>
> 1. In the user experience section, the user is expected to provide supply
> the UUID for each voter. I'm assuming this is the directory.id coming from
> KIP-858. I thought it was generated by the format comand automatically? It
> seems like we will need a way to specify it explicitly.

Thanks for highlighting this, Jason. Leaving aside the bootstrapping paradox 
you just mentioned, I think it's extremely unfriendly to ask people to paste 
auto-generated directory IDs into the kafka-storage format command. We need a 
better solution for this -- one that doesn't require huge amounts of manual 
work for people setting up clusters.

I think this is closely related to the problem of taking existing clusters 
(which effectively don't have directory IDs for controllers) and converting 
them to the new world. I think we can agree that a software upgrade process 
that requires manually configuring 3 UUIDs (or whatever) using painstaking 
manual commands on each controller node is a nonstarter.

Overall, I do not think we should add any new flags to "kafka-storage.sh 
format".

One approach might be to support both DIRID-less and DIRID-ful modes of 
operation of the quorum. Empty logs would be considered DIRID-less, and would 
remain so until the active controller decided to write out the record 
establishing the IDs. (And if the MV was low enough, it would just never do 
that). Given that we have to support DIRID-less mode anyway, this seems viable.

> 2. Do we need the AddVoters and RemoveVoters control records? Perhaps the
> VotersRecord is sufficient since changes to the voter set will be rare.

Agreed. Voter changes should be extremely rare. Listing the full set makes 
debugging easier as well.

> 4. Should ReplicaUuid in FetchRequest be a tagged field? It seems like a
> lot of overhead for all consumer fetches.

+1

best,
Colin

>
> On Mon, Jan 8, 2024 at 10:13 AM José Armando García Sancio
>  wrote:
>
>> Hi all,
>>
>> KIP-853: KRaft Controller Membership Changes is ready for another
>> round of discussion.
>>
>> There was a previous discussion thread at
>> https://lists.apache.org/thread/zb5l1fsqw9vj25zkmtnrk6xm7q3dkm1v
>>
>> I have changed the KIP quite a bit since that discussion. The core
>> idea is still the same. I changed some of the details to be consistent
>> with some of the protocol changes to Kafka since the original KIP. I
>> also added a section that better describes the feature's UX.
>>
>> KIP: https://cwiki.apache.org/confluence/x/nyH1D
>>
>> Thanks. Your feedback is greatly appreciated!
>> --
>> -José
>>

Re: [PR] Add get support page [kafka-site]

[via GitHub]

showuon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446975450


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.

Review Comment:
   I know @ableegoldman is trying to list all helpful resources here, but I 
don't think adding `Confluent Community Slack` or `StackOverflow` is 
appropriate here. `Confluent Community Slack` is run by Confluent, so it is not 
a neutral recourse. About the `StackOverflow`, I think if we list 
`StackOverflow` here, does that imply `us...@apache.kafka.org` or 
`d...@apache.kafka.org` are not useful? I think, as a community, we should 
recommend users to ask/answer questions in the community, instead of in the 3rd 
party site. 



##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Kafka is a widely used project. As such, several 
companies have built products and services around Kafka.
+   This page is dedicated to providing descriptions of 
those offerings and links to more information.
+   Companies are definitely encouraged to update this page 
directly or send a mail to the Kafka PMC with a description
+   of your offerings, and we can update the page. The 
products and services listed on this page are provided for
+   information use only to our users. The Kafka PMC does 
not endorse or recommend any of the products or services
+   on this page. See below for information about what is 
appropriate to add to the page.
+   
+
+   
+   https://www.yupiik.com; 
target="_blank">Yupiik contributes and commits to many Apache projects. 
Provides consulting, training and support

Review Comment:
   I think you should start a discussion thread to ask if we "should" have the 
"Commercial support" section first. I agree with @ableegoldman that it's weird 
the Apache Kafka site contains any commercial content.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-15735) KRaft support in SaslMultiMechanismConsumerTest

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-15735.
---
Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaft support in SaslMultiMechanismConsumerTest
> ---
>
> Key: KAFKA-15735
> URL: https://issues.apache.org/jira/browse/KAFKA-15735
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Assignee: Sanskar Jhajharia
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in SaslMultiMechanismConsumerTest in 
> core/src/test/scala/integration/kafka/api/SaslMultiMechanismConsumerTest.scala
>  need to be updated to support KRaft
> 45 : def testMultipleBrokerMechanisms(): Unit = {
> Scanned 94 lines. Found 0 KRaft tests out of 1 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Colin McCabe]

On Tue, Jan 9, 2024, at 18:56, Proven Provenzano wrote:
> Hi folks,
>
> Thank you for the questions.
>
> Let me clarify about reorder first. The reorder of unstable metadata
> versions should be infrequent.

Why does it need to be infrequent? We should be able to reorder unstable 
metadata versions as often as we like. There are no guarantees about unstable 
MVs.

> The time you reorder is when a feature that
> requires a higher metadata version to enable becomes "production ready" and
> the features with unstable metadata versions less than the new stable one
> are moved to metadata versions greater than the new stable feature. When we
> reorder, we are always allocating a new MV and we are never reusing an
> existing MV even if it was also unstable. This way a developer upgrading
> their environment with a specific unstable MV might see existing
> functionality stop working but they won't see new MV dependent
> functionality magically appear. The feature set for a given unstable MV
> version can only decrease with reordering.

If you use an unstable MV, you probably won't be able to upgrade your software. 
Because whenever something changes, you'll probably get serialization 
exceptions being thrown inside the controller. Fatal ones.

Given that this is true, there's no reason to have special rules about what we 
can and can't do with unstable MVs. We can do anything.

>
> How do we define "production ready" and when should we bump
> LATEST_PRODUCTION? I would like to define it to be the point where the
> feature is code complete with tests and the KIP for it is approved. However
> even with this definition if the feature later develops a major issue it
> could still block future features until the issue is fixed which is what we
> are trying to avoid here. We could be much more formal about this and let
> the release manager for a release define what is stable for a given release
> and then do the bump just after the branch is created on the branch. When
> an RC candidate is accepted, the bump would be backported. I would like to
> hear other ideas here.
>

Yeah, it's an interesting question. Overall, I think developers should define 
when a feature is production ready.

The question to ask is, "are you ready to take this feature to production in 
your workplace?" I think most developers do have a sense of this. Obviously 
bugs and mistakes can happen, but I think this standard would avoid most of the 
issues that we're trying to avoid by having unstable MVs in the first place.

ELR is a good example. Nobody would have said that it was production ready in 
3.7 ... hence it belonged (and still belongs) in an unstable MV, until that 
changes (hopefully soon :) )

best,
Colin

> --Proven
>
> On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe  wrote:
>
>> Hi Justine,
>>
>> Yes, this is an important point to clarify. Proven can comment more, but
>> my understanding is that we can do anything to unstable metadata versions.
>> Reorder them, delete them, change them in any other way. There are no
>> stability guarantees. If the current text is unclear let's add more
>> examples of what we can do (which is anything) :)
>>
>> best,
>> Colin
>>
>>
>> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote:
>> > Hey Colin,
>> >
>> > I had some offline discussions with Proven previously and it seems like
>> he
>> > said something different so I'm glad I brought it up here.
>> >
>> > Let's clarify if we are ok with reordering unstable metadata versions :)
>> >
>> > Justine
>> >
>> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe  wrote:
>> >
>> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
>> >> > Hey all,
>> >> >
>> >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata
>> >> > version. Is this something we should do as soon as the feature is
>> >> complete
>> >> > or something we do when we are releasing kafka. When is the time we
>> >> abandon
>> >> > a MV so that other features can be unblocked?
>> >>
>> >> Hi Justine,
>> >>
>> >> Thanks for reviewing.
>> >>
>> >> The idea is that you should bump LATEST_PRODUCTION when you want to
>> take a
>> >> feature to production. That could mean deploying it internally
>> somewhere to
>> >> production, or doing an Apache release that lets everyone deploy the
>> thing
>> >> to production.
>> >>
>> >> Not in production? No need to care about this. Make any changes you
>> like.
>> >>
>> >> As a corollary, we should keep the LATEST_PRODUCTION version as low as
>> it
>> >> can be. If you haven't tested the feature, don't freeze it in stone yet.
>> >>
>> >> >
>> >> > I am just considering a feature that may end up missing a release. It
>> >> seems
>> >> > like maybe that MV would block future metadata versions until we
>> decide
>> >> the
>> >> > feature won't make the cut. From that point, all "ready" features
>> should
>> >> be
>> >> > able to be released.
>> >>
>> >> The intention is the opposite. A feature in an unstable metadata version
>> >> doesn't block 

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446950508


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Kafka is a widely used project. As such, several 
companies have built products and services around Kafka.
+   This page is dedicated to providing descriptions of 
those offerings and links to more information.
+   Companies are definitely encouraged to update this page 
directly or send a mail to the Kafka PMC with a description
+   of your offerings, and we can update the page. The 
products and services listed on this page are provided for
+   information use only to our users. The Kafka PMC does 
not endorse or recommend any of the products or services
+   on this page. See below for information about what is 
appropriate to add to the page.
+   
+
+   
+   https://www.yupiik.com; 
target="_blank">Yupiik contributes and commits to many Apache projects. 
Provides consulting, training and support

Review Comment:
   @ableegoldman do you think we can start a thread on the mailing list to ask 
which company want to be add in the commercial list support?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446949226


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Kafka is a widely used project. As such, several 
companies have built products and services around Kafka.
+   This page is dedicated to providing descriptions of 
those offerings and links to more information.
+   Companies are definitely encouraged to update this page 
directly or send a mail to the Kafka PMC with a description
+   of your offerings, and we can update the page. The 
products and services listed on this page are provided for
+   information use only to our users. The Kafka PMC does 
not endorse or recommend any of the products or services
+   on this page. See below for information about what is 
appropriate to add to the page.
+   
+
+   
+   https://www.yupiik.com; 
target="_blank">Yupiik contributes and commits to many Apache projects. 
Provides consulting, training and support

Review Comment:
   I added the first company because I don't know the other and like it's 
mention in the text, company are encouraged to add themselves by providing a 
PR. At the beginning we need to start with one company but if people want to 
add other companies in this PR with comments, I can add them to start with more 
than one before merging.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446947444


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review

Review Comment:
   Ok, I will update to be more specific about the issue tracker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446946324


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.

Review Comment:
   There is an existing ASF Slack and a Kafka channel so why not using it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446946944


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.

Review Comment:
   I will mention and add a link to StackOverFlow.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

PR https://github.com/apache/kafka/pull/14531

[James Olsen]

Would appreciate it if someone would take a look at 
https://github.com/apache/kafka/pull/14531.
A fairly simple enhancement to allow me to stop using a deprecated API.

Thanks.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2555

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #56

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Proven Provenzano]

Hi folks,

Thank you for the questions.

Let me clarify about reorder first. The reorder of unstable metadata
versions should be infrequent. The time you reorder is when a feature that
requires a higher metadata version to enable becomes "production ready" and
the features with unstable metadata versions less than the new stable one
are moved to metadata versions greater than the new stable feature. When we
reorder, we are always allocating a new MV and we are never reusing an
existing MV even if it was also unstable. This way a developer upgrading
their environment with a specific unstable MV might see existing
functionality stop working but they won't see new MV dependent
functionality magically appear. The feature set for a given unstable MV
version can only decrease with reordering.

How do we define "production ready" and when should we bump
LATEST_PRODUCTION? I would like to define it to be the point where the
feature is code complete with tests and the KIP for it is approved. However
even with this definition if the feature later develops a major issue it
could still block future features until the issue is fixed which is what we
are trying to avoid here. We could be much more formal about this and let
the release manager for a release define what is stable for a given release
and then do the bump just after the branch is created on the branch. When
an RC candidate is accepted, the bump would be backported. I would like to
hear other ideas here.

--Proven

On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe  wrote:

> Hi Justine,
>
> Yes, this is an important point to clarify. Proven can comment more, but
> my understanding is that we can do anything to unstable metadata versions.
> Reorder them, delete them, change them in any other way. There are no
> stability guarantees. If the current text is unclear let's add more
> examples of what we can do (which is anything) :)
>
> best,
> Colin
>
>
> On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote:
> > Hey Colin,
> >
> > I had some offline discussions with Proven previously and it seems like
> he
> > said something different so I'm glad I brought it up here.
> >
> > Let's clarify if we are ok with reordering unstable metadata versions :)
> >
> > Justine
> >
> > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe  wrote:
> >
> >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
> >> > Hey all,
> >> >
> >> > I was wondering how often we plan to update LATEST_PRODUCTION metadata
> >> > version. Is this something we should do as soon as the feature is
> >> complete
> >> > or something we do when we are releasing kafka. When is the time we
> >> abandon
> >> > a MV so that other features can be unblocked?
> >>
> >> Hi Justine,
> >>
> >> Thanks for reviewing.
> >>
> >> The idea is that you should bump LATEST_PRODUCTION when you want to
> take a
> >> feature to production. That could mean deploying it internally
> somewhere to
> >> production, or doing an Apache release that lets everyone deploy the
> thing
> >> to production.
> >>
> >> Not in production? No need to care about this. Make any changes you
> like.
> >>
> >> As a corollary, we should keep the LATEST_PRODUCTION version as low as
> it
> >> can be. If you haven't tested the feature, don't freeze it in stone yet.
> >>
> >> >
> >> > I am just considering a feature that may end up missing a release. It
> >> seems
> >> > like maybe that MV would block future metadata versions until we
> decide
> >> the
> >> > feature won't make the cut. From that point, all "ready" features
> should
> >> be
> >> > able to be released.
> >>
> >> The intention is the opposite. A feature in an unstable metadata version
> >> doesn't block anything. You can always move a feature from one unstable
> >> metadata version to another if the feature starts taking too long to
> finish.
> >>
> >> > I'm also wondering if the KIP should include some information about
> how a
> >> > metadata should be abandoned. Maybe there is a specific message to
> write
> >> in
> >> > the file? So folks who were maybe waiting on that version know they
> can
> >> > release their feature?
> >> >
> >> > I am also assuming that we don't shift all the waiting metadata
> versions
> >> > when we abandon a version, but it would be good to clarify and
> include in
> >> > the KIP.
> >>
> >> I'm not sure what you mean by abandoning a version. We never abandon a
> >> version once it's stable.
> >>
> >> Unstable versions can change. I wouldn't describe this as "abandonment",
> >> just the MV changing prior to release.
> >>
> >> In a similar way, the contents of the 3.7 branch will change up until
> >> 3.7.0 is released. Once it gets released, it's never unreleased. We just
> >> move on to 3.7.1. Same thing here.
> >>
> >> best,
> >> Colin
> >>
> >> >
> >> > Thanks,
> >> >
> >> > Justine
> >> >
> >> > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe 
> wrote:
> >> >
> >> >> Hi Proven,
> >> >>
> >> >> Thanks for the KIP. I think there is a need for this capability, for
> >> 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Jason Gustafson]

Hi Jose,

Thanks for the KIP! A few initial questions below:

1. In the user experience section, the user is expected to provide supply
the UUID for each voter. I'm assuming this is the directory.id coming from
KIP-858. I thought it was generated by the format comand automatically? It
seems like we will need a way to specify it explicitly.
2. Do we need the AddVoters and RemoveVoters control records? Perhaps the
VotersRecord is sufficient since changes to the voter set will be rare.
3. Why does UpdateVoter need to commit after every leader change?
4. Should ReplicaUuid in FetchRequest be a tagged field? It seems like a
lot of overhead for all consumer fetches.
5. I was looking for the correctness conditions when changing the voter
set. I recalled something about always taking the voter set from the log
even if it is not yet known to be committed. Do you have those details
documented?


-Jason

On Mon, Jan 8, 2024 at 10:13 AM José Armando García Sancio
 wrote:

> Hi all,
>
> KIP-853: KRaft Controller Membership Changes is ready for another
> round of discussion.
>
> There was a previous discussion thread at
> https://lists.apache.org/thread/zb5l1fsqw9vj25zkmtnrk6xm7q3dkm1v
>
> I have changed the KIP quite a bit since that discussion. The core
> idea is still the same. I changed some of the details to be consistent
> with some of the protocol changes to Kafka since the original KIP. I
> also added a section that better describes the feature's UX.
>
> KIP: https://cwiki.apache.org/confluence/x/nyH1D
>
> Thanks. Your feedback is greatly appreciated!
> --
> -José
>

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Colin McCabe]

Hi José,

Thanks for the revisions. I'm really excited to see this going forward for 
Kafka 3.8.

One important piece of feedback that a lot of people have given me is that they 
really want auto-formatting in KRaft mode. In other words, they want to start 
up a process and just have it do the right thing, without having to run a 
special command like "kafka-storage.sh format" to set up the storage 
directories.

One reason why users want auto-formatting is that ZK mode had it. Of course, ZK 
mode's auto-formatting is not safe. It can lead to data loss since it breaks 
the replication invariant that brokers never lose data after ACKing it. But 
hardly any users are aware of this. All they know is that they want things to 
work like they did in ZK mode.

Another reason why users want auto-formatting is that it makes it easier to 
integrate Kafka into systems like Kubernetes, Ansible, Puppet, and so forth. 
These systems generally let the administrator set a "desired state." They then 
take a look at the "actual state" and manipulate it until it matches the 
desired state.

These process management systems tend to be oriented around spinning up a new 
process or dropping in a new config file. They don't like to make RPCs or 
invoke command-line tools. Of course it's POSSIBLE to make them do this, but it 
feels awkward, and it's extra work. Even worse, it's work that the integrators 
tend to get wrong. Most of them don't understand why naively re-formatting a 
controller storage directory every time it looks empty is a bad idea.

In other words, If we don't implement auto-formatting in Kafka, the integrators 
will re-invent it outside Kafka. And they'll almost certainly do it incorrectly 
in a way that may cause metadata loss. So I really do think we should get this 
right in Kafka itself.

If we can run through a few scenarios here:

1. restarting a controller with an empty storage directory

The controller can contact the quorum to get the cluster ID and current MV. If 
the MV doesn't support quorum reconfiguration, it can bail out. Otherwise, it 
can re-format itself with a random directory ID. It can then remove (ID, 
OLD_DIR_ID) from the quorum, and add (ID, NEW_DIR_ID) to the quorum.

I think this can all be done automatically without user intervention. If the 
remove / add steps fail (because the quorum is down, for example), then of 
course we can just log an exception and bail out.

2. restarting a broker with an empty storage directory

The broker can contact the quorum to get the cluster ID and current MV. If the 
MV doesn't support directory IDs, we can bail out. Otherwise, it can reformat 
itself with a random directory ID and start up. Its old replicas will be 
correctly treated as gone due to the JBOD logic.

3. restarting a controller with a damaged metadata directory

I think we can just bail out if the storage directory doesn't look right. Empty 
is OK. Damaged is not.

4. Bringing up a totally new cluster

I think we need at least one controller node to be formatted, so that we can 
decide what metadata version to use. Perhaps we should even require a quorum of 
controller nodes to be explicitly formatted (aka, in practice, people just 
format them all).

5. Removing a controller

I think in this case, we can have an explicit command. This is similar to the 
broker case, where we have the "kafka-cluster.sh unregister" command.

best,
Colin


On Mon, Jan 8, 2024, at 10:13, José Armando García Sancio wrote:
> Hi all,
>
> KIP-853: KRaft Controller Membership Changes is ready for another
> round of discussion.
>
> There was a previous discussion thread at
> https://lists.apache.org/thread/zb5l1fsqw9vj25zkmtnrk6xm7q3dkm1v
>
> I have changed the KIP quite a bit since that discussion. The core
> idea is still the same. I changed some of the details to be consistent
> with some of the protocol changes to Kafka since the original KIP. I
> also added a section that better describes the feature's UX.
>
> KIP: https://cwiki.apache.org/confluence/x/nyH1D
>
> Thanks. Your feedback is greatly appreciated!
> -- 
> -José

[jira] [Resolved] (KAFKA-16094) BrokerRegistrationRequest.logDirs field must be ignorable

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-16094.
--
Fix Version/s: 3.7.0
   Resolution: Fixed

> BrokerRegistrationRequest.logDirs field must be ignorable
> -
>
> Key: KAFKA-16094
> URL: https://issues.apache.org/jira/browse/KAFKA-16094
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Blocker
> Fix For: 3.7.0
>
>
> 3.7 brokers must be able to register with 3.6 and earlier controllers. So 
> this means that the logDirs field must be ignorable (aka, not sent) if the 
> highest BrokerRegistrationRequest version we can negotiate is older than v2.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Colin McCabe]

KAFKA-16094 has been fixed and backported to 3.7.

Colin


On Mon, Jan 8, 2024, at 14:52, Colin McCabe wrote:
> On an unrelated note, I found a blocker bug related to upgrades from 
> 3.6 (and earlier) to 3.7.
>
> The JIRA is here: 
>   https://issues.apache.org/jira/browse/KAFKA-16094
>
> Fix here:
>   https://github.com/apache/kafka/pull/15153
>
> best,
> Colin
>
>
> On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote:
>> Hi Ismael,
>>
>> I wasn't aware of that. If we are required to publish all modules, then 
>> this is working as intended.
>>
>> I am a bit curious if we've discussed why we need to publish the server 
>> modules to Sonatype. Is there a discussion about the pros and cons of 
>> this somewhere?
>>
>> regards,
>> Colin
>>
>> On Mon, Jan 8, 2024, at 14:09, Ismael Juma wrote:
>>> All modules are published to Sonatype - that's a requirement. You may be
>>> missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`.
>>>
>>> Ismael
>>>
>>> On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe  wrote:
>>>
 Hi Ismael,

 It seems like both the metadata gradle module and the server-common module
 are getting published to Sonatype as separate artifacts, unless I'm
 misunderstanding something. Example:

 https://central.sonatype.com/search?q=kafka-server-common

 I don't see kafka-core getting published, but maybe other private
 server-side gradle modules are getting published.

 This seems bad. Is there a reason to publish modules that are only used by
 the server on Sonatype?

 best,
 Colin


 On Mon, Jan 8, 2024, at 12:50, Ismael Juma wrote:
 > Hi Colin,
 >
 > I think you may have misunderstood what they mean by gradle metadata -
 it's
 > not the Kafka metadata module.
 >
 > Ismael
 >
 > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe  wrote:
 >
 >> Oops, hit send too soon. I see that #15127 was already merged. So we
 >> should no longer be publishing :metadata as part of the clients
 artifacts,
 >> right?
 >>
 >> thanks,
 >> Colin
 >>
 >>
 >> On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote:
 >> > Hi Apporv,
 >> >
 >> > Please remove the metadata module from any artifacts published for
 >> > clients. It is only used by the server.
 >> >
 >> > best,
 >> > Colin
 >> >
 >> >
 >> > On Sun, Jan 7, 2024, at 03:04, Apoorv Mittal wrote:
 >> >> Hi Colin,
 >> >> Thanks for the response. The only reason for asking the question of
 >> >> publishing the metadata is because that's present in previous client
 >> >> releases. For more context, the description of PR
 >> >>  holds the details and
 >> waiting
 >> >> for the confirmation there prior to the merge.
 >> >>
 >> >> Regards,
 >> >> Apoorv Mittal
 >> >> +44 7721681581
 >> >>
 >> >>
 >> >> On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe 
 >> wrote:
 >> >>
 >> >>> metadata is an internal gradle module. It is not used by clients.
 So I
 >> >>> don't see why you would want to publish it (unless I'm
 misunderstanding
 >> >>> something).
 >> >>>
 >> >>> best,
 >> >>> Colin
 >> >>>
 >> >>>
 >> >>> On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote:
 >> >>> > Thanks for reporting the blockers, folks. Good job finding.
 >> >>> >
 >> >>> > I have one ask - can anybody with Gradle expertise help review
 this
 >> small
 >> >>> > PR? https://github.com/apache/kafka/pull/15127 (+1, -1)
 >> >>> > In particular, we are wondering whether we need to publish module
 >> >>> metadata
 >> >>> > as part of the gradle publishing process.
 >> >>> >
 >> >>> >
 >> >>> > On Fri, Jan 5, 2024 at 3:56 PM Proven Provenzano
 >> >>> >  wrote:
 >> >>> >
 >> >>> >> We have potentially one more blocker
 >> >>> >> https://issues.apache.org/jira/browse/KAFKA-16082 which might
 >> cause a
 >> >>> data
 >> >>> >> loss scenario with JBOD in KRaft.
 >> >>> >> Initial analysis thought this is a problem and further review
 looks
 >> >>> like it
 >> >>> >> isn't but we are continuing to dig into the issue to ensure that
 it
 >> >>> isn't.
 >> >>> >> We would request feedback on the bug from anyone who is familiar
 >> with
 >> >>> this
 >> >>> >> code.
 >> >>> >>
 >> >>> >> --Proven
 >> >>> >>
 >> >>> >
 >> >>> >
 >> >>> > --
 >> >>> > Best,
 >> >>> > Stanislav
 >> >>>
 >>

Re: [PR] Add get support page [kafka-site]

[via GitHub]

rmannibucau commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446679772


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Kafka is a widely used project. As such, several 
companies have built products and services around Kafka.
+   This page is dedicated to providing descriptions of 
those offerings and links to more information.
+   Companies are definitely encouraged to update this page 
directly or send a mail to the Kafka PMC with a description
+   of your offerings, and we can update the page. The 
products and services listed on this page are provided for
+   information use only to our users. The Kafka PMC does 
not endorse or recommend any of the products or services
+   on this page. See below for information about what is 
appropriate to add to the page.
+   
+
+   
+   https://www.yupiik.com; 
target="_blank">Yupiik contributes and commits to many Apache projects. 
Provides consulting, training and support

Review Comment:
   An alternative to reaching out (which feels like the opposite) is to add an 
edit link (github) and encourage people to pr to add themselves.
   Will make it more open and less closed to a single company so cna be a trade 
off maybe.
   Wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2554

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15946) AsyncKafkaConsumer should retry commits on the application thread instead of auto-retry

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15946.

Fix Version/s: 3.7.0
   (was: 3.8.0)
 Assignee: Lianet Magrans  (was: Kirk True)
   Resolution: Fixed

3.7 includes fix to make sure that only sync commits are retried, with a 
timeout, and async commits are not (just passing failure to the callback). 
There is also a follow ticket https://issues.apache.org/jira/browse/KAFKA-16033

> AsyncKafkaConsumer should retry commits on the application thread instead of 
> auto-retry
> ---
>
> Key: KAFKA-15946
> URL: https://issues.apache.org/jira/browse/KAFKA-15946
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848-client-support
> Fix For: 3.7.0
>
>
> The original design was that the network thread always completes the future 
> whether succeeds or fails.  However, in the current patch, I mis-added 
> auto-retry functionality because commitSync wasn't retrying.  What we should 
> be doing is, the commit sync API should catch the RetriableExceptions and 
> resend another commit until timesout.
>  
> {code:java}
> if (error.exception() instanceof RetriableException) {
> log.warn("OffsetCommit failed on partition {} at offset {}: {}", tp, offset, 
> error.message());
> handleRetriableError(error, response);
> retry(responseTime);  <--- We probably shouldn't do this.
> return;
> } {code}
>  
> {code:java}
> @Override
> public void commitSync(Map offsets, 
> Duration timeout) {
> acquireAndEnsureOpen();
> long commitStart = time.nanoseconds();
> try
> { CompletableFuture commitFuture = commit(offsets, true); <-- we 
> probably should retry here ConsumerUtils.getResult(commitFuture, 
> time.timer(timeout)); }
> finally
> { wakeupTrigger.clearTask(); 
> kafkaConsumerMetrics.recordCommitSync(time.nanoseconds() - commitStart); 
> release(); }
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15967) Fix revocation in reconcilation logic

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15967.

Fix Version/s: 3.7.0
   (was: 3.8.0)
 Assignee: Lianet Magrans
   Resolution: Fixed

> Fix revocation in reconcilation logic
> -
>
> Key: KAFKA-15967
> URL: https://issues.apache.org/jira/browse/KAFKA-15967
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Lucas Brutschy
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: kip-848, kip-848-client-support
> Fix For: 3.7.0
>
>
> Looks like there is a problem in the reconciliation logic.
> We are getting 6 partitions from an HB, we add them to 
> {{{}assignmentReadyToReconcile{}}}. Next HB we get only 4 partitions (2 are 
> revoked), we also add them to {{{}assignmentReadyToReconcile{}}}, but the 2 
> partitions that were supposed to be removed from the assignment are never 
> removed because they are still in {{{}assignmentReadyToReconcile{}}}.
> This was discovered during integration testing of 
> [https://github.com/apache/kafka/pull/14878] - part of the test 
> testRemoteAssignorRange was disabled and should be re-enabled once this is 
> fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Add get support page [kafka-site]

[via GitHub]

ableegoldman commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446628388


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.

Review Comment:
   It might be good to be a bit more specific here. For example most support 
related questions should probably go to the `us...@apache.kafka.org` mailing 
list specifically



##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.

Review Comment:
   Another common source of help is the Confluent Community Slack. Yes, it's 
run by Confluent, but it's free and is fairly active with people, both users 
and devs, from across a wide range of companies. Not sure if we'd need to get 
permission to include that or not.
   
   I also see people ask and get good answers on StackOverflow, though 
generally speaking the users mailing list and Confluent Community Slack are the 
most active in my experience



##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review

Review Comment:
   I feel like this should link to JIRA directly, and also mention it by name 
instead of saying "one of our issue trackers" -- we really only have the one. 
The "contributing" link would be more appropriate for the following sentence, 
which does indeed appear to be about contributing rather than reporting issues 
or "community support". Frankly this section is kind of all over the place but 
I think it's fine as long as we hit on the important notes -- like where to 
report issues



##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as 

[jira] [Resolved] (KAFKA-15872) Investigate autocommit retry logic

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15872.

Fix Version/s: (was: 3.8.0)
   Resolution: Duplicate

> Investigate autocommit retry logic
> --
>
> Key: KAFKA-15872
> URL: https://issues.apache.org/jira/browse/KAFKA-15872
> Project: Kafka
>  Issue Type: Task
>  Components: clients, consumer
>Reporter: Philip Nee
>Priority: Minor
>  Labels: consumer-threading-refactor
>
> This is purely an investigation ticket.
> Currently, we send an autocommit only if there isn't an inflight one; 
> however, this logic might not be correct because I think we should:
>  # expires the request if it is not completed in time
>  # always send an autocommit on the clock



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15455) Add support for OffsetCommit version 9 in consumer

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-15455.

Fix Version/s: 3.7.0
   (was: 3.8.0)
   Resolution: Fixed

> Add support for OffsetCommit version 9 in consumer
> --
>
> Key: KAFKA-15455
> URL: https://issues.apache.org/jira/browse/KAFKA-15455
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: David Jacot
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: kip-848, kip-848-client-support, kip-848-e2e, 
> kip-848-preview
> Fix For: 3.7.0
>
>
> We need to handle the new error codes as specified here:
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/OffsetCommitResponse.json#L46|https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/OffsetCommitRequest.json#L35]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

[Colin McCabe]

Hi Justine,

Yes, this is an important point to clarify. Proven can comment more, but my 
understanding is that we can do anything to unstable metadata versions. Reorder 
them, delete them, change them in any other way. There are no stability 
guarantees. If the current text is unclear let's add more examples of what we 
can do (which is anything) :)

best,
Colin


On Mon, Jan 8, 2024, at 14:18, Justine Olshan wrote:
> Hey Colin,
>
> I had some offline discussions with Proven previously and it seems like he
> said something different so I'm glad I brought it up here.
>
> Let's clarify if we are ok with reordering unstable metadata versions :)
>
> Justine
>
> On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe  wrote:
>
>> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote:
>> > Hey all,
>> >
>> > I was wondering how often we plan to update LATEST_PRODUCTION metadata
>> > version. Is this something we should do as soon as the feature is
>> complete
>> > or something we do when we are releasing kafka. When is the time we
>> abandon
>> > a MV so that other features can be unblocked?
>>
>> Hi Justine,
>>
>> Thanks for reviewing.
>>
>> The idea is that you should bump LATEST_PRODUCTION when you want to take a
>> feature to production. That could mean deploying it internally somewhere to
>> production, or doing an Apache release that lets everyone deploy the thing
>> to production.
>>
>> Not in production? No need to care about this. Make any changes you like.
>>
>> As a corollary, we should keep the LATEST_PRODUCTION version as low as it
>> can be. If you haven't tested the feature, don't freeze it in stone yet.
>>
>> >
>> > I am just considering a feature that may end up missing a release. It
>> seems
>> > like maybe that MV would block future metadata versions until we decide
>> the
>> > feature won't make the cut. From that point, all "ready" features should
>> be
>> > able to be released.
>>
>> The intention is the opposite. A feature in an unstable metadata version
>> doesn't block anything. You can always move a feature from one unstable
>> metadata version to another if the feature starts taking too long to finish.
>>
>> > I'm also wondering if the KIP should include some information about how a
>> > metadata should be abandoned. Maybe there is a specific message to write
>> in
>> > the file? So folks who were maybe waiting on that version know they can
>> > release their feature?
>> >
>> > I am also assuming that we don't shift all the waiting metadata versions
>> > when we abandon a version, but it would be good to clarify and include in
>> > the KIP.
>>
>> I'm not sure what you mean by abandoning a version. We never abandon a
>> version once it's stable.
>>
>> Unstable versions can change. I wouldn't describe this as "abandonment",
>> just the MV changing prior to release.
>>
>> In a similar way, the contents of the 3.7 branch will change up until
>> 3.7.0 is released. Once it gets released, it's never unreleased. We just
>> move on to 3.7.1. Same thing here.
>>
>> best,
>> Colin
>>
>> >
>> > Thanks,
>> >
>> > Justine
>> >
>> > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe  wrote:
>> >
>> >> Hi Proven,
>> >>
>> >> Thanks for the KIP. I think there is a need for this capability, for
>> those
>> >> of us who deploy from trunk (or branches dervied from trunk).
>> >>
>> >> With regard to "unstable.metadata.versions.enable": is this going to be
>> a
>> >> documented configuration, or an internal one? I am guessing we want it
>> to
>> >> be documented, so that users can use it. If we do, we should probably
>> also
>> >> very prominently warn that THIS WILL BREAK UPGRADES FOR YOUR CLUSTER.
>> That
>> >> includes logging an ERROR message on startup, etc.
>> >>
>> >> It would be good to document if a release can go out that contains
>> "future
>> >> MVs" that are unstable. Like can we make a 3.8 release that contains
>> >> IBP_4_0_IV0 in MetadataVersion.java, as an unstable future MV?
>> Personally I
>> >> think the answer should be "yes," but with the usual caveats. When the
>> >> actual 4.0 comes out, the unstable 4.0 MV that shipped in 3.8 probably
>> >> won't work, and you won't be able to upgrade. (It was unstable, we told
>> you
>> >> not to use it.)
>> >>
>> >> best,
>> >> Colin
>> >>
>> >>
>> >> On Fri, Jan 5, 2024, at 07:32, Proven Provenzano wrote:
>> >> > Hey folks,
>> >> >
>> >> > I am starting a discussion thread for managing unstable metadata
>> >> > versions
>> >> > in Apache Kafka.
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1014%3A+Managing+Unstable+Metadata+Versions+in+Apache+Kafka
>> >> >
>> >> > This KIP is actually already implemented in 3.7 with PR
>> >> > https://github.com/apache/kafka/pull/14860.
>> >> > I have created this KIP to explain the motivation and how managing
>> >> Metadata
>> >> > Versions is expected to work.
>> >> > Comments are greatly appreciated as this process can always be
>> improved.
>> >> >
>> >> > --
>> >> > --Proven
>> >>
>>

Re: Kafka trunk test & build stability

[Colin McCabe]

Sorry, but to put it bluntly, the current build setup isn't good enough at 
partial rebuilds that build caching would make sense. All Kafka devs have had 
the experience of needing to clean the build directory in order to get a valid 
build. The scala code esspecially seems to have this issue.

regards,
Colin


On Tue, Jan 2, 2024, at 07:00, Nick Telford wrote:
> Addendum: I've opened a PR with what I believe are the changes necessary to
> enable Remote Build Caching, if you choose to go that route:
> https://github.com/apache/kafka/pull/15109
>
> On Tue, 2 Jan 2024 at 14:31, Nick Telford  wrote:
>
>> Hi everyone,
>>
>> Regarding building a "dependency graph"... Gradle already has this
>> information, albeit fairly coarse-grained. You might be able to get some
>> considerable improvement by configuring the Gradle Remote Build Cache. It
>> looks like it's currently disabled explicitly:
>> https://github.com/apache/kafka/blob/trunk/settings.gradle#L46
>>
>> The trick is to have trunk builds write to the cache, and PR builds only
>> read from it. This way, any PR based on trunk should be able to cache not
>> only the compilation, but also the tests from dependent modules that
>> haven't changed (e.g. for a PR that only touches the connect/streams
>> modules).
>>
>> This would probably be preferable to having to hand-maintain some
>> rules/dependency graph in the CI configuration, and it's quite
>> straight-forward to configure.
>>
>> Bonus points if the Remote Build Cache is readable publicly, enabling
>> contributors to benefit from it locally.
>>
>> Regards,
>> Nick
>>
>> On Tue, 2 Jan 2024 at 13:00, Lucas Brutschy 
>> wrote:
>>
>>> Thanks for all the work that has already been done on this in the past
>>> days!
>>>
>>> Have we considered running our test suite with
>>> -XX:+HeapDumpOnOutOfMemoryError and uploading the heap dumps as
>>> Jenkins build artifacts? This could speed up debugging. Even if we
>>> store them only for a day and do it only for trunk, I think it could
>>> be worth it. The heap dumps shouldn't contain any secrets, and I
>>> checked with the ASF infra team, and they are not concerned about the
>>> additional disk usage.
>>>
>>> Cheers,
>>> Lucas
>>>
>>> On Wed, Dec 27, 2023 at 2:25 PM Divij Vaidya 
>>> wrote:
>>> >
>>> > I have started to perform an analysis of the OOM at
>>> > https://issues.apache.org/jira/browse/KAFKA-16052. Please feel free to
>>> > contribute to the investigation.
>>> >
>>> > --
>>> > Divij Vaidya
>>> >
>>> >
>>> >
>>> > On Wed, Dec 27, 2023 at 1:23 AM Justine Olshan
>>> 
>>> > wrote:
>>> >
>>> > > I am still seeing quite a few OOM errors in the builds and I was
>>> curious if
>>> > > folks had any ideas on how to identify the cause and fix the issue. I
>>> was
>>> > > looking in gradle enterprise and found some info about memory usage,
>>> but
>>> > > nothing detailed enough to help figure the issue out.
>>> > >
>>> > > OOMs sometimes fail the build immediately and in other cases I see it
>>> get
>>> > > stuck for 8 hours. (See
>>> > >
>>> > >
>>> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka/detail/trunk/2508/pipeline/12
>>> > > )
>>> > >
>>> > > I appreciate all the work folks are doing here and I will continue to
>>> try
>>> > > to help as best as I can.
>>> > >
>>> > > Justine
>>> > >
>>> > > On Tue, Dec 26, 2023 at 1:04 PM David Arthur
>>> > >  wrote:
>>> > >
>>> > > > S2. We’ve looked into this before, and it wasn’t possible at the
>>> time
>>> > > with
>>> > > > JUnit. We commonly set a timeout on each test class (especially
>>> > > integration
>>> > > > tests). It is probably worth looking at this again and seeing if
>>> > > something
>>> > > > has changed with JUnit (or our usage of it) that would allow a
>>> global
>>> > > > timeout.
>>> > > >
>>> > > >
>>> > > > S3. Dedicated infra sounds nice, if we can get it. It would at least
>>> > > remove
>>> > > > some variability between the builds, and hopefully eliminate the
>>> > > > infra/setup class of failures.
>>> > > >
>>> > > >
>>> > > > S4. Running tests for what has changed sounds nice, but I think it
>>> is
>>> > > risky
>>> > > > to implement broadly. As Sophie mentioned, there are probably some
>>> lines
>>> > > we
>>> > > > could draw where we feel confident that only running a subset of
>>> tests is
>>> > > > safe. As a start, we could probably work towards skipping CI for
>>> non-code
>>> > > > PRs.
>>> > > >
>>> > > >
>>> > > > ---
>>> > > >
>>> > > >
>>> > > > As an aside, I experimented with build caching and running affected
>>> > > tests a
>>> > > > few months ago. I used the opportunity to play with Github Actions,
>>> and I
>>> > > > quite liked it. Here’s the workflow I used:
>>> > > >
>>> https://github.com/mumrah/kafka/blob/trunk/.github/workflows/push.yml. I
>>> > > > was trying to see if we could use a build cache to reduce the
>>> compilation
>>> > > > time on PRs. A nightly/periodic job would build trunk and populate a
>>> > > Gradle
>>> > > > build cache. 

[jira] [Created] (KAFKA-16106) group size counters do not reflect the actual sizes when operations fail

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-16106:


 Summary: group size counters do not reflect the actual sizes when 
operations fail
 Key: KAFKA-16106
 URL: https://issues.apache.org/jira/browse/KAFKA-16106
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim
Assignee: Jeff Kim


An expire-group-metadata operation generates tombstone records, updates the 
`groups` state and decrements group size counters, then performs a write to the 
log. If there is a __consumer_offsets partition reassignment, this operation 
fails. The `groups` state is reverted to an earlier snapshot but classic group 
size counters are not. This begins an inconsistency between the metrics and the 
actual groups size. This applies to all unsuccessful write operations that 
alter the `groups` state.

 

The issue is exacerbated because the expire group metadata operation is retried 
possibly indefinitely.

 

The solution to this is to make the counters also a timeline data structure 
(TimelineLong) so that in the event of a failed write operation we revert the 
counters as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-15250) DefaultBackgroundThread is running tight loop

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True reopened KAFKA-15250:
---
  Assignee: Kirk True  (was: Philip Nee)

> DefaultBackgroundThread is running tight loop
> -
>
> Key: KAFKA-15250
> URL: https://issues.apache.org/jira/browse/KAFKA-15250
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Major
>  Labels: consumer-threading-refactor
>
> The DefaultBackgroundThread is running tight loops and wasting CPU cycles.  I 
> think we need to reexamine the timeout pass to networkclientDelegate.poll.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15941) Flaky test: shouldRestoreNullRecord() – org.apache.kafka.streams.integration.RestoreIntegrationTest

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-15941.

Resolution: Cannot Reproduce

> Flaky test: shouldRestoreNullRecord() – 
> org.apache.kafka.streams.integration.RestoreIntegrationTest
> ---
>
> Key: KAFKA-15941
> URL: https://issues.apache.org/jira/browse/KAFKA-15941
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Reporter: Apoorv Mittal
>Priority: Major
>  Labels: flaky-test
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/
> {code:java}
> org.opentest4j.AssertionFailedError: Condition not met within timeout 6. 
> Did not receive all [KeyValue(2, \x00\x00\x00)] records from topic output 
> (got []) ==> expected:  but was: 
> Stacktraceorg.opentest4j.AssertionFailedError: Condition not met 
> within timeout 6. Did not receive all [KeyValue(2, \x00\x00\x00)] records 
> from topic output (got []) ==> expected:  but was:   at 
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at 
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)   
>   at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)  at 
> org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331) 
>at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328) 
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312) at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302) at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:878)
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:827)
>  at 
> org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:790)
>  at 
> org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreNullRecord(RestoreIntegrationTest.java:244)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16099) Handle timeouts for AsyncKafkaConsumer.commitSync

[Lianet Magrans (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lianet Magrans resolved KAFKA-16099.

Fix Version/s: 3.7.0
   Resolution: Fixed

> Handle timeouts for AsyncKafkaConsumer.commitSync
> -
>
> Key: KAFKA-16099
> URL: https://issues.apache.org/jira/browse/KAFKA-16099
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Andrew Schofield
>Priority: Major
> Fix For: 3.7.0
>
>
> The handling of synchronous offset commits in the background thread does not 
> observe the caller's timeout. In the situation that a commit request needs to 
> be retried, the retries should not extend beyond the caller's timeout. The 
> CommitApplicationEvent should contain the timeout and not continue beyond 
> that time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Justine Olshan]

Thanks Raman.

+1 (binding) from me as well.

Justine

On Tue, Jan 9, 2024 at 10:12 AM Jun Rao  wrote:

> Hi, Raman,
>
> Thanks for the KIP. +1 from me.
>
> Jun
>
> On Tue, Dec 26, 2023 at 11:32 AM Raman Verma 
> wrote:
>
> > I would like to start a Vote on KIP-994
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
> >
>

[jira] [Created] (KAFKA-16105) Reassignment of tiered topics is failing due to RemoteStorageException

[Anatolii Popov (Jira)]

Anatolii Popov created KAFKA-16105:
--

 Summary: Reassignment of tiered topics is failing due to 
RemoteStorageException
 Key: KAFKA-16105
 URL: https://issues.apache.org/jira/browse/KAFKA-16105
 Project: Kafka
  Issue Type: Bug
  Components: Tiered-Storage
Reporter: Anatolii Popov


When partition reassignment is happening for a tiered topic in most of the 
cases it's stuck with RemoteStorageException's on follower nodes saying that it 
can not construct remote log auxilary state:

 
{code:java}
[2024-01-09 08:34:02,899] ERROR [ReplicaFetcher replicaId=7, leaderId=6, 
fetcherId=2] Error building remote log auxiliary state for test-24 
(kafka.server.ReplicaFetcherThread)
                                         
org.apache.kafka.server.log.remote.storage.RemoteStorageException: Couldn't 
build the state from remote store for partition: test-24, currentLeaderEpoch: 
8, leaderLocalLogStartOffset: 209, leaderLogStartOffset: 0, epoch: 0 as the 
previous remote log segment metadata was not found
                                                 at 
kafka.server.ReplicaFetcherTierStateMachine.buildRemoteLogAuxState(ReplicaFetcherTierStateMachine.java:259)
                                                 at 
kafka.server.ReplicaFetcherTierStateMachine.start(ReplicaFetcherTierStateMachine.java:106)
                                                 at 
kafka.server.AbstractFetcherThread.handleOffsetsMovedToTieredStorage(AbstractFetcherThread.scala:762)
                                                 at 
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:413)
                                                 at 
scala.Option.foreach(Option.scala:437)
                                                 at 
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:332)
                                                 at 
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:331)
                                                 at 
kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
                                                 at 
scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:407)
                                                 at 
scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:403)
                                                 at 
scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:321)
                                                 at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:331)
                                                 at 
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
                                                 at 
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
                                                 at 
scala.Option.foreach(Option.scala:437)
                                                 at 
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
                                                 at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112)
                                                 at 
kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98)
                                                 at 
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130)
 {code}
 

Scenario:

A cluster of 3 nodes with a single topic with 30 partitions. All partitions 
have tiered segments.
Adding 3 more nodes to the cluster and making a reassignment to move all the 
data to new nodes.
Behavior:
For most of the partitions reassignment is happening smoothly.
For some of the partitions when a new node starts to get assignments it reads 
__remote_log_metadata topic and tries to initialize the metadata cache on 
records with COPY_SEGMENT_STARTED. If it's reading such a message for the 
partition before the partition was assigned to this specific node it ignores 
the message, so skips the cache initialization and marks the partition as 
assigned. So reassignment is stuck since COPY_SEGMENT_STARTED is never properly 
processed.

Expected behavior:
The partitions should not be marked as assigned before the cache is initialized 
to be able to re-read COPY_SEGMENT_STARTED message and initialize the cache.

 

Some notes:
This is most probably happening when there are messages in a single metadata 
partition and the order of the messages does not correspond to the order of 
assignment. So the follower reads the COPY_SEGMENT_STARTED 

Re: [VOTE] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Jun Rao]

Hi, Raman,

Thanks for the KIP. +1 from me.

Jun

On Tue, Dec 26, 2023 at 11:32 AM Raman Verma 
wrote:

> I would like to start a Vote on KIP-994
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2553

[Apache Jenkins Server]

See 

Re: [PR] Add get support page [kafka-site]

[via GitHub]

fpapon commented on PR #577:
URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883454200

   @C0urante good catch! Sorry for the copy/paste error, it's ok now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

rmannibucau commented on PR #577:
URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883451784

   @C0urante yes, this is a small typo @fpapon is aware of but the proposal 
follows ASF guidelines and the fix is trivial (before or after the merge) so a 
clear plus for Kafka community from the last year requests I saw, but you are 
right, we should have highlighted it more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

jbonofre commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446363532


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Camel is a widely used project. As such, several 
companies have built products and services around Camel.

Review Comment:
   It should be Kafka here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

C0urante commented on code in PR #577:
URL: https://github.com/apache/kafka-site/pull/577#discussion_r1446362361


##
support.html:
##
@@ -0,0 +1,54 @@
+
+
+
+
+   
+   
+   Support
+   Community support
+
+   
+   This is an open source project so the amount of time 
community members have available to help resolve your issue
+   is often limited as help is provided on a volunteer 
basis from
+   https://www.apache.org/foundation/how-it-works/#hats; 
target="_blank">individuals.
+   However, it is free, and you are often able to discuss 
issues with the developers who actually wrote the code
+   you are running.
+   
+   
+   If you want community support then please https://kafka.apache.org/contact;>contact us.
+   If you’re fairly certain you’re hitting a bug please 
report it via one of our
+   https://kafka.apache.org/contributing;>issue 
trackers. Be sure to review
+   these pages carefully as they contain tips and tricks 
about working with Apache communities in general and
+   Apache Kafka in particular.
+   
+
+   Commercial support
+   
+   Apache Camel is a widely used project. As such, several 
companies have built products and services around Camel.
+   This page is dedicated to providing descriptions of 
those offerings and links to more information.
+   Companies are definitely encouraged to update this page 
directly or send a mail to the Camel PMC with a description
+   of your offerings, and we can update the page. The 
products and services listed on this page are provided for
+   information use only to our users. The Camel PMC does 
not endorse or recommend any of the products or services
+   on this page. See below for information about what is 
appropriate to add to the page.

Review Comment:
   Wrong project.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

jbonofre commented on PR #577:
URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883435388

   @C0urante yup it's a copy paste from Camel. I already asked @fpapon to 
update the page to mention Kafka instead of Camel.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add get support page [kafka-site]

[via GitHub]

C0urante commented on PR #577:
URL: https://github.com/apache/kafka-site/pull/577#issuecomment-1883431881

   @jbonofre @rmannibucau did you actually review this PR? Part of it is 
copy+pasted with no changes from the [Apache Camel 
docs](https://github.com/apache/camel/blob/74b6e1994a33fe54c3f5a5ad2acc5849148dbb16/docs/user-manual/modules/ROOT/pages/commercial-camel-offerings.adoc#commercial-camel-offerings).
 I would hope that anyone who actually looked at the diff would have spotted 
that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Add get support page [kafka-site]

[via GitHub]

fpapon opened a new pull request, #577:
URL: https://github.com/apache/kafka-site/pull/577

   Proposal to add a `get support` page to describe how community and 
commercial support is working, like we have in many other Apache projects.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #55

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 458989 lines...]
> Task :storage:storage-api:compileTestJava
> Task :storage:storage-api:testClasses
> Task :server:compileTestJava
> Task :server:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.7/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."

> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.7/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :core:compileScala
> Task :clients:javadocJar
> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :streams:generateMetadataFileForMavenJavaPublication
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:javadocJar
> Task :streams:srcJar
> Task :streams:processTestResources UP-TO-DATE
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 5m 3s
95 actionable tasks: 41 executed, 54 up-to-date

Publishing build scan...
https://ge.apache.org/s/z6vduqww7x6c6

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_3.7/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

[Chris Egerton]

Thanks for the KIP! +1 (binding)

On Mon, Jan 8, 2024 at 9:35 AM Ashwin  wrote:

> Hi All,
>
> I would like to start  a vote on KIP-995.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors
>
> Discussion thread -
> https://lists.apache.org/thread/msorbr63scglf4484yq764v7klsj7c4j
>
> Thanks!
>
> Ashwin
>

[jira] [Created] (KAFKA-16104) Enable additional PlaintextConsumerTest tests for new consumer

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16104:


 Summary: Enable additional PlaintextConsumerTest tests for new 
consumer
 Key: KAFKA-16104
 URL: https://issues.apache.org/jira/browse/KAFKA-16104
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Andrew Schofield


It should be possible to enable:
* testAutoCommitOnClose
* testAutoCommitOnCloseAfterWakeup
* testExpandingTopicSubscriptions
* testShrinkingTopicSubscriptions
* testMultiConsumerSessionTimeoutOnClose (KAFKA-16011 has been fixed)
* testAutoCommitOnRebalance
* testPerPartitionLeadMetricsCleanUpWithSubscribe
* testPerPartitionLagMetricsCleanUpWithSubscribe
* testStaticConsumerDetectsNewPartitionCreatedAfterRestart



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16103) Review client logic for triggering offset commit callbacks

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16103:
--

 Summary: Review client logic for triggering offset commit callbacks
 Key: KAFKA-16103
 URL: https://issues.apache.org/jira/browse/KAFKA-16103
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans
Assignee: Lianet Magrans


Review logic for triggering commit callbacks, ensuring that all callbacks are 
triggered before returning from commitSync



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16102) about DynamicListenerConfig, the dynamic modification of the listener's port or IP does not take effect.

[Jialun Peng (Jira)]

Jialun Peng created KAFKA-16102:
---

 Summary: about DynamicListenerConfig, the dynamic modification of 
the listener's port or IP does not take effect.
 Key: KAFKA-16102
 URL: https://issues.apache.org/jira/browse/KAFKA-16102
 Project: Kafka
  Issue Type: Bug
  Components: config
Affects Versions: 3.6.0
 Environment: Must be present in any environment.
Reporter: Jialun Peng
Assignee: Jialun Peng
 Fix For: 3.8.0


When I dynamically modify the parameters related to Kafka listeners, such as 
changing the IP or port value of a listener, the dynamic parameters under the 
corresponding path in ZooKeeper are updated. However, in reality, the 
modification of the IP or port for the corresponding listener does not take 
effect. This phenomenon consistently occurs. And there is a slight improvement 
as the error "Security protocol cannot be updated for existing listener" will 
be eliminated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15982) Move GenericGroup state metrics to `GroupCoordinatorMetricsShard`

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15982.
-
Resolution: Duplicate

Done in KAFKA-15870.

> Move GenericGroup state metrics to `GroupCoordinatorMetricsShard`
> -
>
> Key: KAFKA-15982
> URL: https://issues.apache.org/jira/browse/KAFKA-15982
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> Currently, the generic group state metrics exist inside 
> `GroupCoordinatorMetrics` as global metrics. This causes issues as during 
> unload, we need to traverse through all groups and decrement the group size 
> counters. 
> Move the generic group state metrics to the shard level so that when a 
> partition is unloaded we automatically remove the counter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14949) Add Streams upgrade tests from AK 3.4

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-14949.
-
Resolution: Fixed

> Add Streams upgrade tests from AK 3.4
> -
>
> Key: KAFKA-14949
> URL: https://issues.apache.org/jira/browse/KAFKA-14949
> Project: Kafka
>  Issue Type: Task
>  Components: streams, system tests
>Reporter: Victoria Xia
>Assignee: Mickael Maison
>Priority: Critical
>
> Streams upgrade tests currently only test upgrading from 3.3 and earlier 
> versions 
> ([link|https://github.com/apache/kafka/blob/056657d84d84e116ffc9460872945b4d2b479ff3/tests/kafkatest/tests/streams/streams_application_upgrade_test.py#L30]).
>  We should add 3.4 as an "upgrade_from" version into these tests, in light of 
> the upcoming 3.5 release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15741) KRaft support in DescribeConsumerGroupTest

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15741.

Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaft support in DescribeConsumerGroupTest
> --
>
> Key: KAFKA-15741
> URL: https://issues.apache.org/jira/browse/KAFKA-15741
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Assignee: Zihao Lin
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in DescribeConsumerGroupTest in 
> core/src/test/scala/unit/kafka/admin/DescribeConsumerGroupTest.scala need to 
> be updated to support KRaft
> 39 : def testDescribeNonExistingGroup(): Unit = {
> 55 : def testDescribeWithMultipleSubActions(): Unit = {
> 76 : def testDescribeWithStateValue(): Unit = {
> 97 : def testDescribeOffsetsOfNonExistingGroup(): Unit = {
> 113 : def testDescribeMembersOfNonExistingGroup(): Unit = {
> 133 : def testDescribeStateOfNonExistingGroup(): Unit = {
> 151 : def testDescribeExistingGroup(): Unit = {
> 169 : def testDescribeExistingGroups(): Unit = {
> 194 : def testDescribeAllExistingGroups(): Unit = {
> 218 : def testDescribeOffsetsOfExistingGroup(): Unit = {
> 239 : def testDescribeMembersOfExistingGroup(): Unit = {
> 272 : def testDescribeStateOfExistingGroup(): Unit = {
> 291 : def testDescribeStateOfExistingGroupWithRoundRobinAssignor(): Unit = {
> 310 : def testDescribeExistingGroupWithNoMembers(): Unit = {
> 334 : def testDescribeOffsetsOfExistingGroupWithNoMembers(): Unit = {
> 366 : def testDescribeMembersOfExistingGroupWithNoMembers(): Unit = {
> 390 : def testDescribeStateOfExistingGroupWithNoMembers(): Unit = {
> 417 : def testDescribeWithConsumersWithoutAssignedPartitions(): Unit = {
> 436 : def testDescribeOffsetsWithConsumersWithoutAssignedPartitions(): Unit = 
> {
> 455 : def testDescribeMembersWithConsumersWithoutAssignedPartitions(): Unit = 
> {
> 480 : def testDescribeStateWithConsumersWithoutAssignedPartitions(): Unit = {
> 496 : def testDescribeWithMultiPartitionTopicAndMultipleConsumers(): Unit = {
> 517 : def testDescribeOffsetsWithMultiPartitionTopicAndMultipleConsumers(): 
> Unit = {
> 539 : def testDescribeMembersWithMultiPartitionTopicAndMultipleConsumers(): 
> Unit = {
> 565 : def testDescribeStateWithMultiPartitionTopicAndMultipleConsumers(): 
> Unit = {
> 583 : def testDescribeSimpleConsumerGroup(): Unit = {
> 601 : def testDescribeGroupWithShortInitializationTimeout(): Unit = {
> 618 : def testDescribeGroupOffsetsWithShortInitializationTimeout(): Unit = {
> 634 : def testDescribeGroupMembersWithShortInitializationTimeout(): Unit = {
> 652 : def testDescribeGroupStateWithShortInitializationTimeout(): Unit = {
> 668 : def testDescribeWithUnrecognizedNewConsumerOption(): Unit = {
> 674 : def testDescribeNonOffsetCommitGroup(): Unit = {
> Scanned 699 lines. Found 0 KRaft tests out of 32 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15870) Move new group coordinator metrics from Yammer to Metrics

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15870.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Move new group coordinator metrics from Yammer to Metrics
> -
>
> Key: KAFKA-15870
> URL: https://issues.apache.org/jira/browse/KAFKA-15870
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2552

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #54

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16101) Kafka cluster unavailable during KRaft migration rollback procedure

[Paolo Patierno (Jira)]

Paolo Patierno created KAFKA-16101:
--

 Summary: Kafka cluster unavailable during KRaft migration rollback 
procedure
 Key: KAFKA-16101
 URL: https://issues.apache.org/jira/browse/KAFKA-16101
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.6.1
Reporter: Paolo Patierno


Hello,

I was trying the KRaft migration rollback procedure locally and I came across a 
potential bug or anyway a situation where the cluster is not usable/available 
for a certain amount of time.

In order to test the procedure, I start with a one broker and one zookeeper 
node cluster. Then I start the migration with a one KRaft controller node. The 
migration runs fine and it reaches the point of "dual write" state.

>From this point, I try to run the rollback procedure as described in the 
>documentation.

As first step, this involves ...
 * stopping the broker
 * removing the __cluster_metadata folder
 * removing ZooKeeper migration flag and controller(s) related configuration 
from the broker
 * restarting the broker

With the above steps done, the broker starts in ZooKeeper mode (no migration, 
no KRaft controllers knowledge) and it keeps logging the following messages in 
DEBUG:
{code:java}
[2024-01-08 11:51:20,608] DEBUG 
[zk-broker-0-to-controller-forwarding-channel-manager]: Controller isn't 
cached, looking for local metadata changes 
(kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,608] DEBUG 
[zk-broker-0-to-controller-forwarding-channel-manager]: No controller provided, 
retrying after backoff (kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,629] DEBUG 
[zk-broker-0-to-controller-alter-partition-channel-manager]: Controller isn't 
cached, looking for local metadata changes 
(kafka.server.BrokerToControllerRequestThread)
[2024-01-08 11:51:20,629] DEBUG 
[zk-broker-0-to-controller-alter-partition-channel-manager]: No controller 
provided, retrying after backoff (kafka.server.BrokerToControllerRequestThread) 
{code}
What's happening should be clear.

The /controller znode in ZooKeeper still reports the KRaft controller 
(broker.id = 1) as controller. The broker get it from the znode but doesn't 
know how to reach it.

The issue is that until the procedure is complete with the next steps (shutting 
down KRaft controller, deleting /controller znode), the cluster is unusable. 
Any admin or client operation against the broker doesn't work, just hangs, the 
broker doesn't reply.

Imagining this scenario to a more complex one with 10-20-50 brokers and 
partitions' replicas spread across them, when the brokers are rolled one by one 
(in ZK mode) reporting the above error, the topics will become not available 
one after the other, until all brokers are in such a state and nothing can 
work. This is because from a KRaft controller perspective (still running), the 
brokers are not available anymore and the partitions' replicas are out of sync.

Of course, as soon as you complete the rollback procedure, after deleting the 
/controller znode, the brokers are able to elect a new controller among them 
and everything recovers to work.

My first question ... isn't the cluster supposed to work during rollback and 
being always available during the rollback when the procedure is not completed 
yet? Or having the cluster not available is an assumption during the rollback, 
until it's complete?

This "unavailability" time window could be reduced by deleting the /controller 
znode before shutting down the KRaft controllers to allow the brokers electing 
a new controller among them, but in this case, could be a race condition where 
KRaft controllers still running could steal leadership again?

Or is there anything missing in the documentation maybe which is driving to 
this problem?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16100) Consistent handling of timeouts and responses for new consumer ApplicationEvents

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16100:


 Summary: Consistent handling of timeouts and responses for new 
consumer ApplicationEvents
 Key: KAFKA-16100
 URL: https://issues.apache.org/jira/browse/KAFKA-16100
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Reporter: Andrew Schofield


The handling of timeouts and responses for the various kinds of 
ApplicationEvents in the new consumer is not consistent. A small amount of 
refactoring would make the code more maintainable and give consistent behaviour 
for the different requests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16099) Handle timeouts for AsyncKafkaConsumer.commitSync

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-16099:


 Summary: Handle timeouts for AsyncKafkaConsumer.commitSync
 Key: KAFKA-16099
 URL: https://issues.apache.org/jira/browse/KAFKA-16099
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Reporter: Andrew Schofield


The handling of synchronous offset commits in the background thread does not 
observe the caller's timeout. In the situation that a commit request needs to 
be retried, the retries should not extend beyond the caller's timeout. The 
CommitApplicationEvent should contain the timeout and not continue beyond that 
time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14519) Add metrics to the new coordinator

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-14519.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Add metrics to the new coordinator
> --
>
> Key: KAFKA-14519
> URL: https://issues.apache.org/jira/browse/KAFKA-14519
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: Jeff Kim
>Priority: Major
>  Labels: kip-848-preview
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15726) KRaft support in ProduceRequestTest

[Manikumar (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-15726.
---
Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaft support in ProduceRequestTest
> ---
>
> Key: KAFKA-15726
> URL: https://issues.apache.org/jira/browse/KAFKA-15726
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in ProduceRequestTest in 
> core/src/test/scala/unit/kafka/server/ProduceRequestTest.scala need to be 
> updated to support KRaft
> 45 : def testSimpleProduceRequest(): Unit = {
> 82 : def testProduceWithInvalidTimestamp(): Unit = {
> 128 : def testProduceToNonReplica(): Unit = {
> 170 : def testCorruptLz4ProduceRequest(): Unit = {
> 204 : def testZSTDProduceRequest(): Unit = {
> Scanned 253 lines. Found 0 KRaft tests out of 5 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

[Josep Prat]

Hi all,

the vote passes with:

- 4 binding votes: Colin, Chris, Greg and myself
- 1 non-binding vote: Anton

Thanks everyone for the discussion and feedback!

Best,

On Mon, Jan 8, 2024 at 11:08 PM Colin McCabe  wrote:

> Hi Chris,
>
> Based on some of the offline discussions we had, people generally feel
> that major/minor releases have a large enough overhead that they don't want
> them more frequently than every 4 months. (Obviously dot releases are a
> different story) So Josep and I didn't want to raise the issue here.
>
> I also feel that 4 months should be enough (for anyone? :) ) But I'm
> always an optimist.
>
> best,
> Colin
>
>
> On Mon, Jan 8, 2024, at 13:03, Chris Egerton wrote:
> > Hi Colin,
> >
> > The idea isn't to hold up 4.0.0 any longer; in fact, it's the opposite.
> If
> > things slip a bit with 3.8.0 and we take, for example, an extra month to
> > deliver it (or even to cut the branch), I don't think this should
> > necessarily translate to an extra month of latency between now and 4.0.0,
> > given exactly what you mention about the major changes we plan to include
> > in 4.0.0 (which consist more of dropping support for existing things than
> > adding support for new things).
> >
> > If we want to avoid confusion, we could say something like "no later
> than 3
> > to 4 months after the 3.8 branch is created". Frankly though, I think
> it's
> > unnecessary to specify an exact timeline for 4.0 in this KIP, since
> nothing
> > in the proposal actually diverges from the usual time-based release plan
> we
> > follow. The only necessary part seems to be to state that 4.0 will
> directly
> > follow 3.8 (as opposed to 3.9, 3.10, etc.). But perhaps I'm missing
> > something?
> >
> > Cheers,
> >
> > Chris
> >
> > On Mon, Jan 8, 2024 at 2:38 PM Colin McCabe  wrote:
> >
> >> On Mon, Jan 8, 2024, at 09:05, Chris Egerton wrote:
> >> > Hi Josep,
> >> >
> >> > Thanks for the KIP! +1 (binding).
> >> >
> >> > One small nit: I don't think we necessarily have to hold ourselves to
> >> > releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting
> the
> >> > timeline section of the KIP). IMO it's fine to leave some wiggle room
> for
> >> > the 4.0.0 release without codifying a timeline in this KIP. Maybe
> >> something
> >> > like "some time after 3.8 branch is created" would be sufficient?
> >> Anyways,
> >> > not a huge thing, I'm sure we'll all give 4.0.0 the flexibility it
> needs
> >> > with the understanding that this KIP is more focused on 3.8.0 than
> 4.0.0.
> >> >
> >>
> >> Hmm... I don't see any obstacles in the path of releasing 4.0 after the
> >> traditional 4 months of development. Keep in mind, we're removing things
> >> from the code (the ability to support JDK8, ZooKeeper mode, etc.), not
> >> adding things. We already support JDK11 so saying that it's the minimum
> is
> >> a very quick change. Similarly, we already support KRaft so saying that
> >> it's the only mode should be a pretty quick change.
> >>
> >> Also, we added a note that "the timeline is very rough" to KIP-833 and
> it
> >> caued all kinds of confusion. So overall I'd prefer to leave the
> language
> >> about 4.0 unchanged.
> >>
> >> best,
> >> Colin
> >>
> >> >
> >> > Cheers,
> >> >
> >> > Chris
> >> >
> >> > On Mon, Jan 8, 2024 at 11:41 AM Greg Harris
>  >> >
> >> > wrote:
> >> >
> >> >> Thanks Josep for leading the KIP and building consensus on 3.8!
> >> >>
> >> >> +1 (binding)
> >> >>
> >> >> Greg
> >> >>
> >> >> On Sun, Jan 7, 2024 at 11:45 PM Josep Prat
>  >> >
> >> >> wrote:
> >> >> >
> >> >> > Hi all,
> >> >> >
> >> >> > Thanks for your comments,
> >> >> > I reworded some parts of the KIP to express that:
> >> >> > - The KIP is to agree that we need at least one more minor version
> in
> >> the
> >> >> > 3.x series
> >> >> > - Explicitly saying that the list of KIPs is not exhaustive and
> that
> >> if
> >> >> > some are not done, we would need another minor version
> >> >> > - Which are the KIPs/Features the community identified that should
> be
> >> >> > present in a 3.x version so they can safely migrate to a potential
> 4.0
> >> >> > version
> >> >> > - The timeline of 4.0 (starting after branching, not after release)
> >> >> > - Wording is now focusing more on the need to have at least another
> >> minor
> >> >> > release in 3.x to enable and ease migration to a potential 4.0
> version
> >> >> >
> >> >> > I always mention potential in terms of 4.0 as we don't know what
> will
> >> be
> >> >> in
> >> >> > there yet, and this KIP's scope is not meant to define this.
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma 
> >> wrote:
> >> >> >
> >> >> > > I agree with Colin. Also, 4.0 would be started after 3.8 is
> branched
> >> >> (not
> >> >> > > after 3.8.0 is released). The rest looks good.
> >> >> > >
> >> >> > > Thanks!
> >> >> > >
> >> >> > > Ismael
> >> >> > >
> >> >> > > On Fri, Jan 5, 2024 at 11:27 PM Colin McCabe  >
> >> >> wrote:
> >> >> > >
> >> >> > > > 

RE: [DISCUSS] KIP-950: Tiered Storage Disablement

[Doğuşcan Namal]

Hi everyone, any progress on the status of this KIP? Overall looks good to
me but I wonder whether we still need to support it for Zookeeper mode
given that it will be deprecated in the next 3 months.

On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> Hi everyone,
> I would like to start a discussion on KIP-950: Tiered Storage Disablement
(
https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
).
>
> This KIP proposes adding the ability to disable and re-enable tiered
storage on a topic.
>
> Thanks,
> Mehari
>

[jira] [Created] (KAFKA-16098) State updater may attempt to resume a task that is not assigned anymore

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-16098:
--

 Summary: State updater may attempt to resume a task that is not 
assigned anymore
 Key: KAFKA-16098
 URL: https://issues.apache.org/jira/browse/KAFKA-16098
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Lucas Brutschy
 Attachments: streams.log.gz

A long-running soak test brought to light this `IllegalStateException`:
{code:java}
[2024-01-07 08:54:13,688] ERROR [i-0637ca8609f50425f-StreamThread-1] Thread 
encountered an error processing soak test 
(org.apache.kafka.streams.StreamsSoakTest)
java.lang.IllegalStateException: No current assignment for partition 
network-id-repartition-1
    at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367)
    at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.resume(SubscriptionState.java:753)
    at 
org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.resume(LegacyKafkaConsumer.java:963)
    at 
org.apache.kafka.clients.consumer.KafkaConsumer.resume(KafkaConsumer.java:1524)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.transitRestoredTaskToRunning(TaskManager.java:857)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRestoredTasksFromStateUpdater(TaskManager.java:979)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.checkStateUpdater(TaskManager.java:791)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.checkStateUpdater(StreamThread.java:1141)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:949)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:686)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:645)
[2024-01-07 08:54:13,688] ERROR [i-0637ca8609f50425f-StreamThread-1] 
stream-client [i-0637ca8609f50425f] Encountered the following exception during 
processing and sent shutdown request for the entire application. 
(org.apache.kafka.streams.KafkaStreams)
org.apache.kafka.streams.errors.StreamsException: 
java.lang.IllegalStateException: No current assignment for partition 
network-id-repartition-1
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:729)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:645)
Caused by: java.lang.IllegalStateException: No current assignment for partition 
network-id-repartition-1
    at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367)
    at 
org.apache.kafka.clients.consumer.internals.SubscriptionState.resume(SubscriptionState.java:753)
    at 
org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.resume(LegacyKafkaConsumer.java:963)
    at 
org.apache.kafka.clients.consumer.KafkaConsumer.resume(KafkaConsumer.java:1524)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.transitRestoredTaskToRunning(TaskManager.java:857)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRestoredTasksFromStateUpdater(TaskManager.java:979)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.checkStateUpdater(TaskManager.java:791)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.checkStateUpdater(StreamThread.java:1141)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:949)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:686)
    ... 1 more {code}
Log (with some common messages filtered) attached.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16097) State updater removes task without pending action in EOSv2

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-16097:
--

 Summary: State updater removes task without pending action in EOSv2
 Key: KAFKA-16097
 URL: https://issues.apache.org/jira/browse/KAFKA-16097
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Lucas Brutschy


A long-running soak encountered the following exception:

 
{code:java}
[2024-01-08 03:06:00,586] ERROR [i-081c089d2ed054443-StreamThread-3] Thread 
encountered an error processing soak test 
(org.apache.kafka.streams.StreamsSoakTest)
java.lang.IllegalStateException: Got a removed task 1_0 from the state updater 
that is not for recycle, closing, or updating input partitions; this should not 
happen
    at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRemovedTasksFromStateUpdater(TaskManager.java:939)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.checkStateUpdater(TaskManager.java:788)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.checkStateUpdater(StreamThread.java:1141)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:949)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:686)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:645)
[2024-01-08 03:06:00,587] ERROR [i-081c089d2ed054443-StreamThread-3] 
stream-client [i-081c089d2ed054443] Encountered the following exception during 
processing and sent shutdown request for the entire application. 
(org.apache.kafka.streams.KafkaStreams)
org.apache.kafka.streams.errors.StreamsException: 
java.lang.IllegalStateException: Got a removed task 1_0 from the state updater 
that is not for recycle, closing, or updating input partitions; this should not 
happen
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:729)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:645)
Caused by: java.lang.IllegalStateException: Got a removed task 1_0 from the 
state updater that is not for recycle, closing, or updating input partitions; 
this should not happen
    at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRemovedTasksFromStateUpdater(TaskManager.java:939)
    at 
org.apache.kafka.streams.processor.internals.TaskManager.checkStateUpdater(TaskManager.java:788)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.checkStateUpdater(StreamThread.java:1141)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runOnceWithoutProcessingThreads(StreamThread.java:949)
    at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:686)
    ... 1 more{code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16096) Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7) (KIP-1013)

[Ismael Juma (Jira)]

Ismael Juma created KAFKA-16096:
---

 Summary: Drop broker and tools support for Java 11 in Kafka 4.0 
(deprecate in 3.7) (KIP-1013)
 Key: KAFKA-16096
 URL: https://issues.apache.org/jira/browse/KAFKA-16096
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

[Ismael Juma]

Hi all,

The vote passes with:

* 8 binding votes (Justine Olshan, Satish Duggana, Viktor Somogyi-Vass,
Divij Vaidya, Mickael Maison, Greg Harris, David Arthur, myself)
* 1 non-binding vote (Kamal Chandraprakash)

Thanks,
Ismael

On Wed, Jan 3, 2024 at 2:28 AM Ismael Juma  wrote:

> Hi all,
>
> I would like to start a vote on KIP-1013.
>
> As stated in the discussion thread, this KIP was proposed after the KIP
> freeze for Apache Kafka 3.7, but it is purely a documentation update (if we
> decide to adopt it) and I believe it would serve our users best if we
> communicate the deprecation for removal sooner (i.e. 3.7) rather than later
> (i.e. 3.8).
>
> Please take a look and cast your vote.
>
> Link:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
>
> Ismael
>