Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

If we use WithDescendingKeys() to generate a RangeQuery to do the reveseQuery, how do we achieve the methods like withRange, withUpperBound, and withLowerBound only in this method? On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng wrote: > I believe there's no need to introduce a method like

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2253

See Changes: -- [...truncated 315358 lines...] Gradle Test Run :streams:test > Gradle Test Executor 78 > TasksTest >

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

I believe there's no need to introduce a method like WithDescendingKeys(). Instead, we can simply add a reverse flag to RangeQuery. Each method within RangeQuery would then accept an additional parameter. If the reverse is set to true, it would indicate the results should be reversed. Initially,

[jira] [Created] (KAFKA-15536) dynamically resize remoteIndexCache

Luke Chen created KAFKA-15536: - Summary: dynamically resize remoteIndexCache Key: KAFKA-15536 URL: https://issues.apache.org/jira/browse/KAFKA-15536 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Colin, I think in your example "do_unclean_recovery" would need to do different things depending on the strategy. do_unclean_recovery() { if (unclean.recovery.manager.enabled) { if (strategy == Aggressive) use UncleanRecoveryManager(waitLastKnownERL=false) // just inspect logs

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #84

See Changes: -- [...truncated 306677 lines...] streams-5: SMOKE-TEST-CLIENT-CLOSED streams-6: SMOKE-TEST-CLIENT-CLOSED streams-4: SMOKE-TEST-CLIENT-CLOSED streams-1:

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi, Colt, The underlying structure of inMemoryKeyValueStore is treeMap. Sincerely, Hanyu On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng wrote: > Hi Bill, > 1. I will update the KIP in accordance with the PR and synchronize their > future updates. > 2. I will use that name. > 3. you mean add

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi Bill, 1. I will update the KIP in accordance with the PR and synchronize their future updates. 2. I will use that name. 3. you mean add something about ordering at the motivation section? Sincerely, Hanyu On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng wrote: > Hi, Walker, > > 1. I will

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi, Walker, 1. I will update the KIP in accordance with the PR and synchronize their future updates. 2. I will use that name. 3. I'll provide additional details in that section. 4. I intend to utilize rangeQuery to achieve what we're referring to as reverseQuery. In essence, reverseQuery is

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Jun, Thanks for the comments. And thanks to Colin's explanation. 41. The unclean recovery manager may need time to be ready for production. So adding a flag to disable the new feature is in case it has any problems. The config may be deprecated or removed when we are ready. 50. Sorry for the

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Ok, I will change it back to following the code, and update them together. On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson wrote: > Hello Hanyu, > > Looking over your kip things mostly make sense but I have a couple of > comments. > > >1. You have "withDescandingOrder()". I think you mean

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Ok, I just talked about it with Matthias, I will change the text back to following the code, and update them together. Sincerely, Hanyu On Tue, Oct 3, 2023 at 3:49 PM Bill Bejeck wrote: > Hi Hanyu, > > Thanks for the KIP, overall it's looking good, but I have a couple of > comments > >- In

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi Hanyu, Thanks for the KIP, overall it's looking good, but I have a couple of comments - In the “Proposed Changes” section there's a reference to `setReverse()` but the code example has “withDescendingOrder()” so I think the text needs an update to reflect the code example. - I

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

+1 Thanks for the KIP On Tue, Oct 3, 2023 at 12:30 PM David Jacot wrote: > Thanks for the KIP. +1 from me as well. > > Best, > David > > Le mar. 3 oct. 2023 à 20:54, Jun Rao a écrit : > > > Hi, Mayank, > > > > Thanks for the detailed explanation in the KIP. +1 from me. > > > > Jun > > > > On

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Calvin, thanks for the KIP! I'm getting up to speed on the discussion. I had a few questions 57. When is the CleanShutdownFile removed? I think it probably happens after registering with the controller, but it would be good to clarify this. 58. Since the broker epoch comes from the controller,

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

todiaz commented on code in PR #542: URL: https://github.com/apache/kafka-site/pull/542#discussion_r1344810163 ## blog.html: ## @@ -22,6 +22,46 @@ Blog + + + +Apache Kafka

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

todiaz commented on code in PR #542: URL: https://github.com/apache/kafka-site/pull/542#discussion_r1344810163 ## blog.html: ## @@ -22,6 +22,46 @@ Blog + + + +Apache Kafka

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

On Tue, Oct 3, 2023, at 10:49, Jun Rao wrote: > Hi, Calvin, > > Thanks for the update KIP. A few more comments. > > 41. Why would a user choose the option to select a random replica as the > leader instead of using unclean.recovery.strateg=Aggressive? It seems that > the latter is strictly better?

Re: [PR] MINOR: Added missing Kafka Broker docs for metrics.jmx.(include|exclude) configs [kafka-site]

Dionakra closed pull request #517: MINOR: Added missing Kafka Broker docs for metrics.jmx.(include|exclude) configs URL: https://github.com/apache/kafka-site/pull/517 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hello Hanyu, Looking over your kip things mostly make sense but I have a couple of comments. 1. You have "withDescandingOrder()". I think you mean "descending" :) Also there are still a few places in the do where its called "setReverse" 2. Also I like "WithDescendingKeys()" better

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

Hi Yash, Thanks for the in-depth discussion! Continuations here: 1. Regarding delimiters (dots vs. underscores), we use dots in connector configs for all runtime-recognized properties, including but not limited to connector.class, tasks.max, key.converter, key.converter.*, and transforms.*.type.

[DISCUSS] KIP-988 Streams Standby Task Update Listener

Hi all, We would like to propose a small KIP to improve the ability of Streams apps to monitor the progress of their standby tasks through a callback interface. We have a nearly-working implementation on our fork and are curious for feedback.

Re: [VOTE] 3.6.0 RC2

Hi, Thank you all for participating in the 3.6.0 release candidate vote threads. This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes. +1 votes PMC Members: * Luke Chen * Bill Bejeck * Chris Egerton * Justine Olshan Community: * Jakub Scholz * Federico Valeri * Divij Vaidya * Kamal

Re: [VOTE] 3.6.0 RC2

Hi, Thank you all for participating in the 3.6.0 release candidate vote threads. This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes. +1 votes PMC Members: * Luke Chen * Bill Bejeck * Chris Egerton * Justine Olshan Community: * Jakub Scholz * Federico Valeri * Divij Vaidya * Kamal

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

Thanks for the KIP. +1 from me as well. Best, David Le mar. 3 oct. 2023 à 20:54, Jun Rao a écrit : > Hi, Mayank, > > Thanks for the detailed explanation in the KIP. +1 from me. > > Jun > > On Wed, Sep 27, 2023 at 4:39 AM Mayank Shekhar Narula < > mayanks.nar...@gmail.com> wrote: > > > Reviving

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2252

See

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hello Hanyu, Thank you for the KIP. I agree with Matthias' proposal to keep the naming convention consistent with KIP-969. I favor the `.withDescendingKeys()` name. I am curious about one thing. RocksDB guarantees that records returned during a range scan are lexicographically ordered by the

[jira] [Created] (KAFKA-15535) Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.

Satish Duggana created KAFKA-15535: -- Summary: Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property. Key: KAFKA-15535 URL: https://issues.apache.org/jira/browse/KAFKA-15535

[jira] [Created] (KAFKA-15534) Propagate client response time when timeout to the request handler

Philip Nee created KAFKA-15534: -- Summary: Propagate client response time when timeout to the request handler Key: KAFKA-15534 URL: https://issues.apache.org/jira/browse/KAFKA-15534 Project: Kafka

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

Hi, Mayank, Thanks for the detailed explanation in the KIP. +1 from me. Jun On Wed, Sep 27, 2023 at 4:39 AM Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > Reviving this thread, as the discussion thread has been updated. > > On Fri, Jul 28, 2023 at 11:29 AM Mayank Shekhar Narula < >

Re: Permission to Create KIP

Hey Colt, You should be all set. Looking forward to the KIP! Greg On Tue, Oct 3, 2023 at 11:37 AM Colt McNealy wrote: > > Hello there, > > Could I please have access to create a Wiki page? A team member and I would > like to jointly propose a small KIP. > > JIRA id: coltmcnealy-lh > > Thank

Permission to Create KIP

Hello there, Could I please have access to create a Wiki page? A team member and I would like to jointly propose a small KIP. JIRA id: coltmcnealy-lh Thank you, Colt McNealy *Founder, LittleHorse.dev*

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

ok, I will update it. Thank you Matthias Sincerely, Hanyu On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax wrote: > Thanks for the KIP Hanyu! > > > I took a quick look and it think the proposal makes sense overall. > > A few comments about how to structure the KIP. > > As you propose to not

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Thanks for the KIP Hanyu! I took a quick look and it think the proposal makes sense overall. A few comments about how to structure the KIP. As you propose to not add `ReverseRangQuery` class, the code example should go into "Rejected Alternatives" section, not in the "Proposed Changes"

[jira] [Created] (KAFKA-15533) Ensure HeartbeatRequestManager only send out some fields once

Philip Nee created KAFKA-15533: -- Summary: Ensure HeartbeatRequestManager only send out some fields once Key: KAFKA-15533 URL: https://issues.apache.org/jira/browse/KAFKA-15533 Project: Kafka

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi, Calvin, Thanks for the update KIP. A few more comments. 41. Why would a user choose the option to select a random replica as the leader instead of using unclean.recovery.strateg=Aggressive? It seems that the latter is strictly better? If that's not the case, could we fold this option under

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hey Alieh thanks for the KIP, Weighing in on the AsOf vs Until debate I think either is fine from a natural language perspective. Personally AsOf makes more sense to me where until gives me the idea that the query is making a change. It's totally a connotative difference and not that important. I

[jira] [Created] (KAFKA-15532) ZkWriteBehindLag should not be reported by inactive controllers

David Arthur created KAFKA-15532: Summary: ZkWriteBehindLag should not be reported by inactive controllers Key: KAFKA-15532 URL: https://issues.apache.org/jira/browse/KAFKA-15532 Project: Kafka

Re: [DISCUSS] KIP-939: Support Participation in 2PC

Hey Artem, Thanks for the KIP. I had a question about epoch bumping. Previously when we send an InitProducerId request on Producer startup, we bump the epoch and abort the transaction. Is it correct to assume that we will still bump the epoch, but just not abort the transaction? If we still bump

[jira] [Created] (KAFKA-15531) Ensure coordinator node is removed upon disconnection exception

Philip Nee created KAFKA-15531: -- Summary: Ensure coordinator node is removed upon disconnection exception Key: KAFKA-15531 URL: https://issues.apache.org/jira/browse/KAFKA-15531 Project: Kafka

Re: [VOTE] 3.6.0 RC2

Thanks folks for following up. Given my previous testing and the results you've provided, I'm +1 (binding) I will also follow up with the non-blocking metrics documentation. Thanks! Justine On Tue, Oct 3, 2023 at 8:17 AM Chris Egerton wrote: > Hi Satish, > > Thanks for running this release! >

Re: [DISCUSS] KIP-714: Client metrics and observability

Hi David, Thanks for your interest in KIP-714. Because this KIP is under development at the same time as KIP-848, it will need to support both the existing KafkaConsumer code and the refactored code being worked on under KIP-848. I’ve updated the Threading section accordingly. Thanks, Andrew >

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

Hi Viktor, Thanks for your questions! I agree, replication is very fundamental in Kafka, so it's been implemented in many different ways by different people. I hope that this is the last implementation we'll need, but every software engineer says that :) GT-1: I think as this KIP is very focused

[DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

https://cwiki.apache.org/confluence/display/KAFKA/KIP-985%3A+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 -- [image: Confluent] Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog]

Re: [VOTE] 3.6.0 RC2

Hi Satish, Thanks for running this release! To verify, I: - Built from source using Java 11 with both: - - the 3.6.0-rc2 tag on GitHub - - the kafka-3.6.0-src.tgz artifact from https://home.apache.org/~satishd/kafka-3.6.0-rc2/ - Checked signatures and checksums - Ran the quickstart using the

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #83

See

Re: [VOTE] 3.6.0 RC2

Hi Justine, Good catch on the "documentation.html" of the generated kafka-site-<>.tgz. It has been missing updates after 3.4. But I think it is a non blocker as the kafka-site repo is updated the documentation with the respective release directories including 3.6. Any pending updates on 3.6 are

Re: [VOTE] 3.6.0 RC2

Thanks Luke for helping on running system tests on RCs and updating the status on this email thread. ~Satish. On Tue, 3 Oct 2023 at 05:04, Luke Chen wrote: > > Hi Justine and all, > > The system test result for 3.6.0 RC2 can be found below. > In short, no failed tests. The flaky tests will pass

Re: [VOTE] 3.6.0 RC2

Thanks Divij for the observations. KAFKA-15483 is added to address the missing metrics related to KRaft and ZK to KRaft migration. Please leave a comment at https://github.com/apache/kafka-site/pull/548 in which related metrics documentation is added.

[jira] [Created] (KAFKA-15530) Add missing documentation of metrics introduced as part of KAFKA-15196

Satish Duggana created KAFKA-15530: -- Summary: Add missing documentation of metrics introduced as part of KAFKA-15196 Key: KAFKA-15530 URL: https://issues.apache.org/jira/browse/KAFKA-15530 Project:

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

Hey Viktor! Thanks for bringing up this use case. I think we can take advantage of Docker volume for this. We can allow users to mount a folder containing the secret files. This folder can then be used to pass secrets to the container. Thanks and regards, Vedarth On Wed, Sep 27, 2023 at 4:39 PM

Re: [VOTE] 3.6.0 RC2

Hi Justine and all, The system test result for 3.6.0 RC2 can be found below. In short, no failed tests. The flaky tests will pass in the 2nd run. https://drive.google.com/drive/folders/1qwIKg-B4CBrswUeo5fBRv65KWpDsGUiS?usp=sharing Thank you. Luke On Tue, Oct 3, 2023 at 7:08 AM Justine Olshan

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2251

See Changes: -- [...truncated 195 lines...] FAILURE: Build failed with an exception. * Where: Build file '/home/jenkins/workspace/Kafka_kafka_trunk/build.gradle' line:

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2250

See Changes: -- [...truncated 195 lines...] * What went wrong: A problem occurred evaluating root project 'kafka'. > Project with path ':storage:api' could not be found

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

Hi Greg, Seems like finding the perfect replication solution is a never ending story for Kafka :). Some general thoughts: GT-1. While as you say it would be good to have some kind of built-in replication in Kafka, we definitely need to understand the problem better to provide a better solution.

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2249

See Changes: -- [...truncated 201 lines...] > git --version # timeout=10 > git --version # 'git version 2.17.1' using GIT_ASKPASS to set credentials ASF Cloudbees

[jira] [Created] (KAFKA-15529) Flaky test ReassignReplicaShrinkTest.executeTieredStorageTest

Divij Vaidya created KAFKA-15529: Summary: Flaky test ReassignReplicaShrinkTest.executeTieredStorageTest Key: KAFKA-15529 URL: https://issues.apache.org/jira/browse/KAFKA-15529 Project: Kafka

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

Hi Hailey, thanks for the KIP. I also agree that the two mutually exclusive args are better. In order to be consistent with the other tools, I would suggest to use --process-role and --node-id (hyphen instead of dot). Can you also update the KIP? On Mon, Oct 2, 2023 at 10:18 PM Hailey Ni wrote:

KRaft Performance Improvements

Do we have any performance test results showing the difference between KRaft vs Zookeeper? The one that I found online is from Redpanda comparing the tail latencies https://redpanda.com/blog/kafka-kraft-vs-redpanda-performance-2023#the-test:-redpanda-23.1-vs.-kafka-3.4.0-with-kraft Can I assume

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

Hi Chris, Thanks for taking a look at this KIP! 1. I chose to go with simply "state" as that exact term is already exposed via some of the existing REST API responses and would be one that users are already familiar with (although admittedly something like "initial_state" wouldn't be much of a