Re: New PR workflow in KAFKA-18748

2025-02-25 Thread Sophie Blee-Goldman
Thanks David! This is awesome, really glad to see this effort to reduce test flakiness. One question -- are tests automatically sorted into these buckets or do we have to manually move them? And if so, how does that work (eg a test in "main" becomes flaky) On Tue, Feb 25, 2025 at 3:20 PM David Ar

[jira] [Created] (KAFKA-18865) Improve KRaft handling of kraft.version

2025-02-25 Thread Jira
José Armando García Sancio created KAFKA-18865: -- Summary: Improve KRaft handling of kraft.version Key: KAFKA-18865 URL: https://issues.apache.org/jira/browse/KAFKA-18865 Project: Kafka

[jira] [Created] (KAFKA-18864) remove the Evolving tag from public interfaces in org.apache.kafka.server.log.remote.storage

2025-02-25 Thread Jun Rao (Jira)
Jun Rao created KAFKA-18864: --- Summary: remove the Evolving tag from public interfaces in org.apache.kafka.server.log.remote.storage Key: KAFKA-18864 URL: https://issues.apache.org/jira/browse/KAFKA-18864 Pr

Re: [VOTE] 4.0.0 RC0

2025-02-25 Thread Ismael Juma
Hi Jun, When it comes to the upgrade documentation, a couple of changes landed after the RC was generated: * https://github.com/apache/kafka/commit/da8f390c4599d7199c4cdf2bb85441146e859b17 * https://github.com/apache/kafka/commit/da3b8e88dc61a1b749895866394cab68410e0eda Regarding the message for

Re: [VOTE] 4.0.0 RC0

2025-02-25 Thread Jun Rao
Hi, David, Thanks for preparing RC0. A few comments on the documentation. 1.5 Upgrading There is no documentation on which previous releases can be upgraded to 4.0. 5.3.3 Old Message Format Should we remove this section since 4.0 only supports the V2 message format? 6.10 Consumer Rebalance Pro

Re: New PR workflow in KAFKA-18748

2025-02-25 Thread David Arthur
> Can we merge the PR if only flaky or new tests fail? I agree with Ismael that new tests must be solid -- no flakiness should be expected when adding a test. Obviously, we will miss things, so we have to tolerate them on trunk (along with environmental flaky factors). If there are existing *unre

Re: [DISCUSS] KIP-1131: Controller-side monitoring for broker shutdown and startup

2025-02-25 Thread Colin McCabe
On Tue, Feb 25, 2025, at 14:40, Colin McCabe wrote: > On Tue, Feb 25, 2025, at 14:12, Kevin Wu wrote: >> Hey Colin, >> >> Thanks for the review. >> >> Regarding the metrics that reflect times: my initial thinking was to indeed >> have these be "soft state", which would be reset when a controller fa

Re: [DISCUSS] KIP-1131: Controller-side monitoring for broker shutdown and startup

2025-02-25 Thread Colin McCabe
On Tue, Feb 25, 2025, at 14:12, Kevin Wu wrote: > Hey Colin, > > Thanks for the review. > > Regarding the metrics that reflect times: my initial thinking was to indeed > have these be "soft state", which would be reset when a controller failover > happens. I'm not sure if it's a big issue if these

Re: [DISCUSS] KIP-1131: Controller-side monitoring for broker shutdown and startup

2025-02-25 Thread Kevin Wu
Hey Colin, Thanks for the review. Regarding the metrics that reflect times: my initial thinking was to indeed have these be "soft state", which would be reset when a controller failover happens. I'm not sure if it's a big issue if these values get reset though, since a controller failover means

RE: Re: [DISCUSS] KIP-1131: Controller-side monitoring for broker shutdown and startup

2025-02-25 Thread Kevin Wu
Hey Colin, Thanks for the review. Regarding the metrics that reflect times: my initial thinking was to indeed have these be "soft state", which would be reset when a controller failover happens. I'm not sure if it's a big issue if these values get reset though, since a controller failover means

Re: kafka producer exception due to TimeoutException:

2025-02-25 Thread Greg Harris
Hi Giri, Thanks for your additional context. Good to see you're running experiments! > How to achieve workload among a different number of producers..could you > suggest java code for this. This is an api calling from dB stored procedure > and they can call any number of times this api with mess

Re: [VOTE] 4.0.0 RC0

2025-02-25 Thread José Armando García Sancio
Hi David and all, I am about to merge https://github.com/apache/kafka/pull/18852 to trunk: https://issues.apache.org/jira/browse/KAFKA-18723. I would like to cherry-pick it to 4.0.0. What do you think? I will also be cherry picking the change back to 3.7.x, 3.8.x and 3.9.x. If it is not included

Re: kafka producer exception due to TimeoutException:

2025-02-25 Thread giri mungi
Hi Greg, Thanks for your insights! I tried increasing the timeouts to: request.timeout.ms = 6 delivery.timeout.ms = 90 However, the issue persists. Some messages are still failing intermittently with a timeout, while others are successfully delivered. props.put(ProducerConfig.ACKS_CONFIG

Re: [PR] MINOR Fix the URL for matomo [kafka-site]

2025-02-25 Thread via GitHub
mumrah commented on PR #663: URL: https://github.com/apache/kafka-site/pull/663#issuecomment-2682948493 Looks like its working! https://analytics.apache.org/index.php?module=CoreHome&action=index&date=yesterday&period=day&idSite=26#?period=day&date=yesterday&idSite=26&category=Dashboard_Dash

Re: [PR] MINOR Fix the URL for matomo [kafka-site]

2025-02-25 Thread via GitHub
mumrah merged PR #663: URL: https://github.com/apache/kafka-site/pull/663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.o

[jira] [Created] (KAFKA-18863) Runtime additions for connector multiversioning.

2025-02-25 Thread Snehashis Pal (Jira)
Snehashis Pal created KAFKA-18863: - Summary: Runtime additions for connector multiversioning. Key: KAFKA-18863 URL: https://issues.apache.org/jira/browse/KAFKA-18863 Project: Kafka Issue Type

Re: kafka producer exception due to TimeoutException:

2025-02-25 Thread Greg Harris
Hi Giri, Since nobody with more experience has answered yet, let me give you my amateur understanding of this error. The TimeoutException will appear whenever the load generation (code calling the Producer) runs faster than all downstream components (Producer, Network, Brokers, etc) can handle. R

Re: [PR] MINOR Fix the URL for matomo [kafka-site]

2025-02-25 Thread via GitHub
mumrah commented on PR #663: URL: https://github.com/apache/kafka-site/pull/663#issuecomment-2682652798 Infra says this looks change looks correct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] MINOR Fix the URL for matomo [kafka-site]

2025-02-25 Thread via GitHub
mumrah opened a new pull request, #663: URL: https://github.com/apache/kafka-site/pull/663 This has been broken for a long time. We should be seeing data in https://analytics.apache.org, but we don't. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [DISCUSS] KIP-1133: AK Documentation and Website in Markdown

2025-02-25 Thread David Arthur
MM1: I suspect Hugo has some way of doing url redirection, but at the very least we can add redirects in our htaccess file https://github.com/apache/kafka-site/blob/asf-site/.htaccess Excluding the "front matter" (intro, quickstart, powered by, etc) I think the main links we need to worry about ar

[jira] [Created] (KAFKA-18862) Review testHeartbeatResponseOnErrorHandling duplicate logic in ShareHeartbeatRequestManagerTest

2025-02-25 Thread Lianet Magrans (Jira)
Lianet Magrans created KAFKA-18862: -- Summary: Review testHeartbeatResponseOnErrorHandling duplicate logic in ShareHeartbeatRequestManagerTest Key: KAFKA-18862 URL: https://issues.apache.org/jira/browse/KAFKA-1886

[DISCUSS] KIP-1136: Make ConsumerGroupMetadata an interface

2025-02-25 Thread Paweł Szymczyk
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1136%3A+Make+ConsumerGroupMetadata+an+interface -- Pozdrawiam Paweł Szymczyk

[jira] [Created] (KAFKA-18861) kafka-streams stuck in a loop of "SyncGroup failed" with an unbalanced assignment

2025-02-25 Thread Jira
Rafał Sumisławski created KAFKA-18861: - Summary: kafka-streams stuck in a loop of "SyncGroup failed" with an unbalanced assignment Key: KAFKA-18861 URL: https://issues.apache.org/jira/browse/KAFKA-18861

Re: [DISCUSS] KIP-1114 Introducing Chunk in Partition

2025-02-25 Thread Viktor Somogyi-Vass
Hi De Gao, I think that the concept of chunking can be a viable idea in itself. The granularity of partitions in some cases may be too coarse, although it can be managed in most cases with repartitioning (adding partitions to increase distribution) and tiered storage (offload old and infrequently

[jira] [Resolved] (KAFKA-17351) Validate compacted topics start offset handling in Share Partition

2025-02-25 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17351. -- Fix Version/s: 4.1.0 Resolution: Fixed > Validate compacted topics start offset

Re: [VOTE] 4.0.0 RC0

2025-02-25 Thread 黃竣陽
Hello David, I've completed the following tests on my local machine: - Built from source using JDK 21 & JDK 17 (Zulu, Amazon, Eclipse) - Executed all unit and integration tests - Ran the quickstart successfully - Tested dynamic voter with 4.0 and 3.9 controllers - Tested 4.0 controller with a 3.9

Support for other OAuth2 grant types in Kafka

2025-02-25 Thread Subra I
Hello All, I see that Kafka by itself supports client credentials as grant type for OAuth2. I see this mentioned in one of the kafka KIP as well: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575 Is there a way to support other grant types as well? I came across the follo

Re: [VOTE] 4.0.0 RC0

2025-02-25 Thread Stanislav Kozlovski
+1 On 2025/02/24 23:07:12 Christopher Shannon wrote: > If the purpose of this is not to vote but just to evaluate the RC then I > think it would be better to not use the [VOTE] tag for the subject as that > is super confusing. > > On Mon, Feb 24, 2025 at 10:45 AM Ismael Juma wrote: > > > Hi Mic

[jira] [Resolved] (KAFKA-17836) Move RackAwareTest to server module

2025-02-25 Thread Chia-Ping Tsai (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-17836. Fix Version/s: 4.1.0 Resolution: Fixed > Move RackAwareTest to server module > ---

[jira] [Created] (KAFKA-18860) Clarify the KRaft missing feature

2025-02-25 Thread Luke Chen (Jira)
Luke Chen created KAFKA-18860: - Summary: Clarify the KRaft missing feature Key: KAFKA-18860 URL: https://issues.apache.org/jira/browse/KAFKA-18860 Project: Kafka Issue Type: Bug Repor

Re: [PR] KAFKA-18330: Update documentation to remove controller deployment limitations [kafka-site]

2025-02-25 Thread via GitHub
chia7712 merged PR #659: URL: https://github.com/apache/kafka-site/pull/659 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache

Re: [DISCUSS] KIP-1133: AK Documentation and Website in Markdown

2025-02-25 Thread Claude Warren, Jr
I have been working on a similar process for Apache RAT and Commons CLI (help module). The help option in RAT 0.17-SNAPSHOT generates the text for the help options partly from the new commons-cli help module and partly from Enums and similar in the code. For the RAT documentation we produce fragm