Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

I agree to (almost) everything what Bruno said. In general, we tend to move away from using getters without "get", recently. So I would keep the "get". This is new to me? Can you elaborate on this point? Why do you think that's the case? I actually did realize (after Walker mentioned it)

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Great discussion! It seems the only open question might be about ordering guarantees? IIRC, we had a discussion about this in the past. Technically (at least from my POV), existing `RangeQuery` does not have a guarantee that data is return in any specific order (not even on a per

[jira] [Resolved] (KAFKA-15547) Thread leak in MirrorMakerConfigTest#testClientConfigProperties

[ https://issues.apache.org/jira/browse/KAFKA-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Mayya resolved KAFKA-15547. Fix Version/s: 3.7.0 Resolution: Fixed > Thread leak in

RE: [DISCUSS] KIP-939: Support Participation in 2PC

Hello Artem, Now that `InitProducerIdRequest` will have an extra parameter (enable2PC), can the client change the value of this parameter during an ongoing transaction. Here is how the transaction coordinator responds to InitProducerId requests according to the current transaction's state. -

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

Hi everyone, Earlier today Colin, Ron, Proven and I had a chat about this work. We discussed several aspects which I’d like to share here. ## A new reserved UUID We'll reserve a third UUID to indicate an unspecified dir, but one that is known to be selected. As opposed to the default

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

Hi Federico, Thanks for your comments. SureI will update the KIP accordingly. Thanks, Hailey On Tue, Oct 3, 2023 at 1:29 AM Federico Valeri wrote: > Hi Hailey, thanks for the KIP. > > I also agree that the two mutually exclusive args are better. In order > to be consistent with the other

Re: [VOTE]KIP-966: Eligible Leader Replicas

Hi, Calvin, Thanks for the KIP. +1 from me too. Jun On Wed, Sep 20, 2023 at 5:28 PM Justine Olshan wrote: > Thanks Calvin. > I think this will be very helpful going forward to minimize data loss. > > +1 from me (binding) > > Justine > > On Wed, Sep 20, 2023 at 3:42 PM Calvin Liu > wrote: > >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #85

See Changes: -- [...truncated 306903 lines...] > Task :connect:api:compileTestJava UP-TO-DATE > Task :connect:api:testClasses UP-TO-DATE > Task :connect:json:jar UP-TO-DATE >

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2257

See Changes: -- [...truncated 421971 lines...] Gradle Test Run :streams:test > Gradle Test Executor 84 > TasksTest >

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi Hanyu, Thanks for the KIP! Seems there are already a lot of good discussions. I only have two comments: 1. Please make it clear in ``` /** * Interactive range query using a lower and upper bound to filter the keys returned. * @param lower The key that specifies the lower bound

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi David, Thanks for the comments. I thought that a new snapshot with the downgraded MV is created in this case. Isn’t it the case? Yes, you are right, a metadata delta will be generated after the MV downgrade. Then the user can start the software downgrade. - Could you also elaborate a

[jira] [Created] (KAFKA-15548) Handling close() properly

Philip Nee created KAFKA-15548: -- Summary: Handling close() properly Key: KAFKA-15548 URL: https://issues.apache.org/jira/browse/KAFKA-15548 Project: Kafka Issue Type: Bug Components:

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2256

See

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

Hi Yash, Looking great! Few more thoughts: 1. (Downgraded to nit) I still prefer dot-delimitation but it's not a blocker; thanks for addressing my concerns about the name of the field and how it may be perceived by users. 2. (Addressed) Thanks for looking into this, and sorry it turned out to

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Calvin, I thought that a new snapshot with the downgraded MV is created in this case. Isn’t it the case? Could you also elaborate a bit more on the reasoning behind adding the limits to the admin RPCs? This is a new pattern in Kafka so it would be good to clear on the motivation. Could you

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Jun, After the MV downgrade, the controller will write in the old version of the PartitionRecord/PartitionChangeRecord. If I understand correctly, it is possible to downgrade the software version if the controller only has to handle old version records. However, the controller will not

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi, Calvin and Justine, Historically, when we change the record format in the log, we don't support software version downgrading. For the record format change in the metadata log, have we thought about forcing the write of the latest metadata records with the old version during MV downgrading?

[jira] [Created] (KAFKA-15547) Thread leak in MirrorMakerConfigTest#testClientConfigProperties

Kalpesh Patel created KAFKA-15547: - Summary: Thread leak in MirrorMakerConfigTest#testClientConfigProperties Key: KAFKA-15547 URL: https://issues.apache.org/jira/browse/KAFKA-15547 Project: Kafka

[PR] KAFKA-15530: Add 3.6 metrics documentation for new transactions metrics [kafka-site]

jolshan opened a new pull request, #555: URL: https://github.com/apache/kafka-site/pull/555 Corresponding kafka-site PR for https://github.com/apache/kafka/pull/14480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[jira] [Created] (KAFKA-15546) Transactions tool duration field confusing for completed transactions

Justine Olshan created KAFKA-15546: -- Summary: Transactions tool duration field confusing for completed transactions Key: KAFKA-15546 URL: https://issues.apache.org/jira/browse/KAFKA-15546 Project:

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Sorry -- not MV but software version. On Wed, Oct 4, 2023 at 9:51 AM Justine Olshan wrote: > Catching up with this discussion. > > I was just curious -- have we had other instances where downgrading MV is > not supported? I think Kafka typically tries to support downgrades, and I > couldn't

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Catching up with this discussion. I was just curious -- have we had other instances where downgrading MV is not supported? I think Kafka typically tries to support downgrades, and I couldn't think of other examples. Thanks, Justine On Wed, Oct 4, 2023 at 9:40 AM Calvin Liu wrote: > Hi Jun, >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi Jun, 54. Marked the software downgrading is not supported. As the old controller will not understand the new PartitionRecord and PartitionChangeRecord. Thanks! On Wed, Oct 4, 2023 at 9:12 AM Jun Rao wrote: > Hi, Calvin, > > Thanks for the reply. Just one more comment. > > 54. It seems that

[jira] [Resolved] (KAFKA-15483) Update metrics documentation for the new metrics implemented as part of KIP-938

[ https://issues.apache.org/jira/browse/KAFKA-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya resolved KAFKA-15483. -- Resolution: Fixed > Update metrics documentation for the new metrics implemented as part of

[jira] [Created] (KAFKA-15545) Update Request metrics in ops.html to reflect all the APIs

Justine Olshan created KAFKA-15545: -- Summary: Update Request metrics in ops.html to reflect all the APIs Key: KAFKA-15545 URL: https://issues.apache.org/jira/browse/KAFKA-15545 Project: Kafka

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

For testing purposes, we previously used a Set to record the results in IQv2StoreIntegrationTest. Let's take an example where we now have two partitions and four key-value pairs: <0,0> in p0, <1,1> in p1, <2,2> in p0, and <3,3> in p1. If we execute withRange(1,3), it will return a Set of <1, 2,

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi, Bruno Thank you for your suggestions, I will update them soon. Sincerely, Hanyu On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng wrote: > Hi, Lucas, > > Thank you for your suggestions. > I will update the KIP and code together. > > Sincerely, > Hanyu > > On Tue, Oct 3, 2023 at 8:16 PM

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi, Lucas, Thank you for your suggestions. I will update the KIP and code together. Sincerely, Hanyu On Tue, Oct 3, 2023 at 8:16 PM Hanyu (Peter) Zheng wrote: > If we use WithDescendingKeys() to generate a RangeQuery to do the > reveseQuery, how do we achieve the methods like withRange,

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

Hey Viktor, Thanks for thinking about Tiered Storage. I'm not so familiar there, so if you could add some of your expectations about how the two features will interact, I would appreciate that. It appears to me that follower-fetch-from-remote is a significant optimization within TS, and so

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

Hi, Calvin, Thanks for the reply. Just one more comment. 54. It seems that downgrading MV is supported. Is downgrading the software version supported? It would be useful to document that. Thanks, Jun On Tue, Oct 3, 2023 at 4:55 PM Artem Livshits wrote: > Hi Colin, > > I think in your

Re: Unsubscribe :

Hey Girish, You may need to confirm the unsubscription with a second email. When I was switching subscription emails, I sent one to the unsubscribe email and then I got a reply. In the reply it asked me to send to a unique email address to confirm. Look for one from dev-h...@kafka.apache.org.

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2255

See

Unsubscribe :

Dear Team I am repeatedly sending email to dev-unsubscr...@kafka.apache.org to unsubscribe this email address of mine from the email notifications received from dev@kafka.apache.org. Could one of you please help me with the correct process? Regards Girish

[jira] [Created] (KAFKA-15544) Enable existing client integration tests for new protocol

Lianet Magrans created KAFKA-15544: -- Summary: Enable existing client integration tests for new protocol Key: KAFKA-15544 URL: https://issues.apache.org/jira/browse/KAFKA-15544 Project: Kafka

[jira] [Created] (KAFKA-15543) Send HB request right after reconciliation completes

Lianet Magrans created KAFKA-15543: -- Summary: Send HB request right after reconciliation completes Key: KAFKA-15543 URL: https://issues.apache.org/jira/browse/KAFKA-15543 Project: Kafka

[jira] [Created] (KAFKA-15542) Release member assignments on errors

Lianet Magrans created KAFKA-15542: -- Summary: Release member assignments on errors Key: KAFKA-15542 URL: https://issues.apache.org/jira/browse/KAFKA-15542 Project: Kafka Issue Type:

[jira] [Resolved] (KAFKA-15438) Review exception caching logic used for reset/validate positions in async consumer

[ https://issues.apache.org/jira/browse/KAFKA-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lianet Magrans resolved KAFKA-15438. Resolution: Fixed > Review exception caching logic used for reset/validate positions in

[DISCUSS] KIP-989: RocksDB Iterator Metrics

Hi everyone, KIP-989 is a small Kafka Streams KIP to add a few new metrics around the creation and use of RocksDB Iterators, to aid users in identifying "Iterator leaks" that could cause applications to leak native memory. Let me know what you think!

[jira] [Created] (KAFKA-15541) RocksDB Iterator Metrics

Nicholas Telford created KAFKA-15541: Summary: RocksDB Iterator Metrics Key: KAFKA-15541 URL: https://issues.apache.org/jira/browse/KAFKA-15541 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-15164) Extract reusable logic from OffsetsForLeaderEpochClient

[ https://issues.apache.org/jira/browse/KAFKA-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lianet Magrans resolved KAFKA-15164. Resolution: Fixed Included in PR https://github.com/apache/kafka/pull/14346 > Extract

[jira] [Created] (KAFKA-15540) Handle heartbeat and assignment release when consumer leaves group

Lianet Magrans created KAFKA-15540: -- Summary: Handle heartbeat and assignment release when consumer leaves group Key: KAFKA-15540 URL: https://issues.apache.org/jira/browse/KAFKA-15540 Project:

[jira] [Created] (KAFKA-15539) Stop fetching while partitions being revoked

Lianet Magrans created KAFKA-15539: -- Summary: Stop fetching while partitions being revoked Key: KAFKA-15539 URL: https://issues.apache.org/jira/browse/KAFKA-15539 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-15538) Resolve regex on client side when using java regex in new consumer

Lianet Magrans created KAFKA-15538: -- Summary: Resolve regex on client side when using java regex in new consumer Key: KAFKA-15538 URL: https://issues.apache.org/jira/browse/KAFKA-15538 Project:

Re: [PR] KAFKA-15483 Add KIP-938 and KIP-866 metrics to ops.html [kafka-site]

mumrah merged PR #548: URL: https://github.com/apache/kafka-site/pull/548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

Hi Greg, Thanks for the answers. I think they all make sense. Another point I realized last evening is that now that tiered storage (TS) is available, it might complicate things with CCR. What I'm thinking of is that if you have multiple clusters in multiple regions, enabling the object

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2254

See Changes: -- [...truncated 315741 lines...] Gradle Test Run :streams:test > Gradle Test Executor 87 > TaskAndActionTest >

[jira] [Created] (KAFKA-15537) Unsafe metadata.version downgrade is not supported

Federico Valeri created KAFKA-15537: --- Summary: Unsafe metadata.version downgrade is not supported Key: KAFKA-15537 URL: https://issues.apache.org/jira/browse/KAFKA-15537 Project: Kafka

[jira] [Resolved] (KAFKA-10199) Separate state restoration into separate threads

[ https://issues.apache.org/jira/browse/KAFKA-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna resolved KAFKA-10199. --- Resolution: Done > Separate state restoration into separate threads >

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi Hanyu, Thanks a lot for the KIP! I agree with Bruno's comments about fluent API / keeping the query classes immutable. Apart from that, on the technical side this is looking good already! Two comments on the KIP itself (not the technical part): - The motivation section could be extended by a

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

mimaison commented on code in PR #547: URL: https://github.com/apache/kafka-site/pull/547#discussion_r1345510589 ## blog.html: ## @@ -22,6 +22,54 @@ Blog + + + +Apache Kafka

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi, Regarding tombstones: As far as I understand, we need to add either a validTo field to VersionedRecord or we need to return tombstones, otherwise the result is not complete, because users could never know a record was deleted at some point before the second non-null value was put. I like

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

Hi Chris, Thanks for the quick follow up and the continued insightful discourse! 1. Fair point on the need to differentiate it from the actual state displayed in the status API, I like the prefix of "initial" to make that differentiation (from your suggested alternatives previously). Regarding

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

raboof commented on code in PR #554: URL: https://github.com/apache/kafka-site/pull/554#discussion_r1345493725 ## project-security.html: ## @@ -35,6 +35,22 @@ Kafka security For a list of security issues fixed in released versions of

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

mimaison commented on code in PR #554: URL: https://github.com/apache/kafka-site/pull/554#discussion_r1345492786 ## project-security.html: ## @@ -35,6 +35,22 @@ Kafka security For a list of security issues fixed in released versions of

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

raboof commented on PR #554: URL: https://github.com/apache/kafka-site/pull/554#issuecomment-1746475927 > Thanks @raboof , can you put a description for detailing the purpose of this pull request ? (updated the commit and PR message) -- This is an automated message from the Apache

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

divijvaidya commented on PR #554: URL: https://github.com/apache/kafka-site/pull/554#issuecomment-1746461093 cc: @mimaison @ijuma @showuon, I would also solicit your opinion on this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

divijvaidya commented on code in PR #554: URL: https://github.com/apache/kafka-site/pull/554#discussion_r1345468001 ## project-security.html: ## @@ -35,6 +35,22 @@ Kafka security For a list of security issues fixed in released versions

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

Summarising, there are 5 binding votes(Luke, Jose, Jun, David, Jason), and 1 non-binding vote(Kirk). With the current status of voting, KIP is accepted. Thanks again to all reviewers and voters. On Wed, Oct 4, 2023 at 9:37 AM Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > Thank

Re: [DISCUSS] KIP-932: Queues for Kafka

I would like to see more explicit discussion of topic retention and share groups. There are a few options here from simple to more sophisticated. There are also topic-level and share-group level options. The simple thing would be to ensure that the SPSO of each share group is bounded by the

Re: [PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

bmscomp commented on PR #554: URL: https://github.com/apache/kafka-site/pull/554#issuecomment-1746451332 Thanks @raboof , can you put a description for detailing the purpose of this pull request ? -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] MINOR: document how we deal with advisories for dependencies [kafka-site]

raboof opened a new pull request, #554: URL: https://github.com/apache/kafka-site/pull/554 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [VOTE] KIP-951: Leader discovery optimisations for the client

Thank you all for your votes, Jun, David, and Jason! On Tue, Oct 3, 2023 at 11:44 PM Jason Gustafson wrote: > +1 Thanks for the KIP > > On Tue, Oct 3, 2023 at 12:30 PM David Jacot wrote: > > > Thanks for the KIP. +1 from me as well. > > > > Best, > > David > > > > Le mar. 3 oct. 2023 à 20:54,

Re: Upgrading from 2.8.0 to 3.5.1

Hi Chester, If I'm not mistaken, what it means is to update the Kafka version on the broker. Broker with version 2.8.0 -> Shutdown -> Replace jars and binaries so it contains Kafka 3.5.1 -> Start it Best, On Wed, Oct 4, 2023 at 9:07 AM Walchester Gaw wrote: > Hello. > > I would like to seek

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

Hi Hanyu, I agree with what others said about having a `withDescendingOrder()` method and about to document how the results are ordered. I would not add a reverse flag and adding a parameter to each method in RangeQuery. This makes the API less fluent and harder to maintain since the flag

Upgrading from 2.8.0 to 3.5.1

Hello. I would like to seek clarification on what it means to "update the code" in your tutorial on how to upgrade. How do I update the code? Which code should I update? [image: image.png] Thanks, Chester