[DISCUSS] KIP-889 Versioned State Stores

Hi everyone, I have a proposal for introducing versioned state stores in Kafka Streams. Versioned state stores are similar to key-value stores except they can store multiple record versions for a single key. This KIP focuses on interfaces only in order to limit the scope of the KIP.

Re: [DISCUSS] Apache Kafka 3.4.0 release

Thanks for the update Rajini, I've added this to the release page since it looks like it will pass but of course if anything changes, just let me know. David, I'm fine with aiming to include KIP-866 in the 3.4 release as well since this seems to be a critical part of the zookeeper

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #117

See Changes: -- [...truncated 501019 lines...] [2022-11-17T01:56:48.900Z] org.apache.kafka.streams.integration.StateDirectoryIntegrationTest > testCleanUpStateDirIfEmpty

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1358

See

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

Hi, Igor, Thanks for the KIP. A few comments below. 10. In the example section, there were two /mnt/d1/meta.properties. One of them should be d2. 11. "When replicas are moved between directories, using the existing AlterReplicaLogDirs RPC, when the future replica has caught up with the current

[jira] [Created] (KAFKA-14397) Idempotent producer may bump epoch and reset sequence numbers prematurely

Jason Gustafson created KAFKA-14397: --- Summary: Idempotent producer may bump epoch and reset sequence numbers prematurely Key: KAFKA-14397 URL: https://issues.apache.org/jira/browse/KAFKA-14397

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1357

See

Re: [DISCUSS] KIP-886 Add Client Producer and Consumer Builders

Hello all, I've gotten great feedback from Knowles here, and from Luke Chen on the jira, so thanks to both so much. This is my first KIP, and I'm pretty new to contributing to kafka, so I'd like to learn a little bit more about the process and the way things usually work. Should I open a PR

[jira] [Created] (KAFKA-14396) Flaky memory leak tests rely on System.gc for correctness

Greg Harris created KAFKA-14396: --- Summary: Flaky memory leak tests rely on System.gc for correctness Key: KAFKA-14396 URL: https://issues.apache.org/jira/browse/KAFKA-14396 Project: Kafka

Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

Hi Mickael, I agree that the new STOPPED state proposed in KIP-875 will improve the connector lifecycle. The changes proposed in this KIP aim to cover the gap where connectors need to actually be deleted, but because the API doesn't provide any hooks, external assets are left lingering where

[jira] [Resolved] (KAFKA-14388) NPE When Retrieving StateStore with new Processor API

[ https://issues.apache.org/jira/browse/KAFKA-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-14388. - Resolution: Fixed Merged PR to trunk > NPE When Retrieving StateStore with new Processor API >

Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

Hi Sagar, Thanks for your feedback! I actually renamed the method from "deleted()" to "destroyed()", which I think conveys the intention more clearly. I can certainly rename it to be 'onDeleted()', although I feel any method named onXXX() belongs to a listener class :) Regarding failure

Re: [DISCUSS] Apache Kafka 3.4.0 release

Hi Sophie, KIP-881 has three binding votes (David Jacot, Jun and me) and one non-binding vote (Maulin). So it is good to go for 3.4.0 if there are no objections until the voting time of 72 hours completes on Friday. Thanks, Rajini On Wed, Nov 16, 2022 at 3:15 PM David Arthur wrote: > Sophie,

[jira] [Resolved] (KAFKA-13785) Support emit final result for windowed aggregation

[ https://issues.apache.org/jira/browse/KAFKA-13785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Li resolved KAFKA-13785. Resolution: Done > Support emit final result for windowed aggregation >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #116

See Changes: -- [...truncated 418979 lines...] [2022-11-16T17:55:55.698Z] [2022-11-16T17:55:55.698Z] org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest

[jira] [Created] (KAFKA-14395) Add config to configure client supplier for KafkaStreams

Hao Li created KAFKA-14395: -- Summary: Add config to configure client supplier for KafkaStreams Key: KAFKA-14395 URL: https://issues.apache.org/jira/browse/KAFKA-14395 Project: Kafka Issue Type:

Re: [VOTE] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

Hi, Rajini, Thanks for the KIP. +1 Jun On Wed, Nov 16, 2022 at 12:10 AM David Jacot wrote: > +1 (binding). Thanks for the KIP, Rajini! > > On Tue, Nov 15, 2022 at 9:26 PM Maulin Vasavada > wrote: > > > > +1 (non-binding). > > > > Makes sense, Rajini. This would be a great addition. > > > >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1356

See

Re: [DISCUSS] Apache Kafka 3.3.2

+1. Thanks for volunteering. -- -José

Re: [DISCUSS] KIP-883: Add delete callback method to Connector API

Hi Hector, Thanks for the KIP. One tricky aspect is that currently there's no real way to stop a connector so to do so people often just delete them temporarily. KIP-875 proposes adding a mechanism to properly stop connectors which should reduce the need to deleting them and avoid doing

Re: [DISCUSS] Apache Kafka 3.3.2

Thanks for volunteering for the release! +1 -Bill On Wed, Nov 16, 2022 at 10:58 AM David Jacot wrote: > +1. Thanks for volunteering! > > On Wed, Nov 16, 2022 at 4:09 PM Chris Egerton > wrote: > > > > Hi all, > > > > I'd like to volunteer to be the release manager for the next bugfix > >

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

Hi, David and Rajini, Thanks for the explanation. It makes sense to me now. Jun On Wed, Nov 16, 2022 at 1:44 AM Rajini Sivaram wrote: > Thanks David, that was my understanding as well. > > Regards, > > Rajini > > On Wed, Nov 16, 2022 at 8:08 AM David Jacot > wrote: > > > Hi Jun, > > > > We

Re: [DISCUSS] Apache Kafka 3.3.2

+1. Thanks for volunteering! On Wed, Nov 16, 2022 at 4:09 PM Chris Egerton wrote: > > Hi all, > > I'd like to volunteer to be the release manager for the next bugfix > release, 3.3.2. > > If there are no objections, I'll send out a release plan by EOD (Eastern > Time) Friday that includes a list

[jira] [Created] (KAFKA-14394) BrokerToControllerChannelManager has 2 separate timeouts

Ron Dagostino created KAFKA-14394: - Summary: BrokerToControllerChannelManager has 2 separate timeouts Key: KAFKA-14394 URL: https://issues.apache.org/jira/browse/KAFKA-14394 Project: Kafka

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #210

See Changes: -- [...truncated 3879 lines...] [2022-11-16T13:25:51.642Z] [2022-11-16T13:25:51.642Z] BUILD FAILED in 6m 29s [2022-11-16T13:25:51.642Z] 208 actionable tasks:

Re: [DISCUSS] Apache Kafka 3.4.0 release

Sophie, the vote for KIP-866 is underway, but there is still some discussion happening. I'm hopeful that the vote can close this week, but it may fall into next week. Can we include this KIP in 3.4? Thanks, David On Tue, Nov 15, 2022 at 6:52 AM Rajini Sivaram wrote: > Hi Sophie, > > I was out

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #135

See Changes: -- [...truncated 295398 lines...] [2022-11-16T15:06:56.030Z] [INFO] Changes detected - recompiling the module! [2022-11-16T15:06:56.030Z] [INFO] Compiling 3

[DISCUSS] Apache Kafka 3.3.2

Hi all, I'd like to volunteer to be the release manager for the next bugfix release, 3.3.2. If there are no objections, I'll send out a release plan by EOD (Eastern Time) Friday that includes a list of all of the fixes we are targeting along with a timeline. Cheers, Chris

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #115

See Changes: -- [...truncated 499842 lines...] [2022-11-16T11:41:01.367Z] org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest >

Re: [DISCUSS] Apache Kafka 3.4.0 release

Thanks Luke and Sophie! Sorry, I had missed out the entry in the KIP page. Regards, Rajini On Wed, Nov 16, 2022 at 3:36 AM Luke Chen wrote: > Hi Sophie, > > Here it is: > KIP-881: > >

[jira] [Resolved] (KAFKA-14334) DelayedFetch purgatory not completed when appending as follower

[ https://issues.apache.org/jira/browse/KAFKA-14334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-14334. - Fix Version/s: 3.4.0 3.3.2 Reviewer: David Jacot Resolution:

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #89

See Changes: -- [...truncated 532943 lines...] [2022-11-16T12:09:50.688Z] [INFO] Parameter: version, Value: 0.1 [2022-11-16T12:09:50.688Z] [INFO] Parameter: package, Value:

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

Thanks David, that was my understanding as well. Regards, Rajini On Wed, Nov 16, 2022 at 8:08 AM David Jacot wrote: > Hi Jun, > > We don't need to bump any RPC requests. The subscription is serialized > (including its version) and included as bytes in the RPCs. > > Best, > David > > On Tue,

Re: [VOTE] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

+1 (binding). Thanks for the KIP, Rajini! On Tue, Nov 15, 2022 at 9:26 PM Maulin Vasavada wrote: > > +1 (non-binding). > > Makes sense, Rajini. This would be a great addition. > > On Tue, Nov 15, 2022 at 10:55 AM Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to start vote on

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

Hi Jun, We don't need to bump any RPC requests. The subscription is serialized (including its version) and included as bytes in the RPCs. Best, David On Tue, Nov 15, 2022 at 11:42 PM Jun Rao wrote: > > Hi, Rajini, > > Thanks for the updated KIP. Just another minor comment. It would be useful >