[DISCUSS] KIP-878: Autoscaling for Statically Partitioned Streams

Hey all, I'd like to propose a new autoscaling feature for Kafka Streams applications which can follow the constraint of static partitioning. For further details please refer to the KIP document:

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #107

See Changes: -- [...truncated 499850 lines...] [2022-10-19T04:15:42.687Z] [2022-10-19T04:15:42.687Z] org.apache.kafka.streams.integration.TableTableJoinIntegrationTest >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1309

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #106

See Changes: -- [...truncated 500263 lines...] [2022-10-19T01:24:51.993Z] org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > testLeftLeft[caching enabled

[jira] [Resolved] (KAFKA-14321) max.compaction.lag.ms is not enforced accurately

[ https://issues.apache.org/jira/browse/KAFKA-14321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-14321. - Resolution: Duplicate This actually duplicates KAFKA-10760. Closing this one. > max.compaction.lag.ms

[jira] [Created] (KAFKA-14321) max.compaction.lag.ms is not enforced accurately

Jun Rao created KAFKA-14321: --- Summary: max.compaction.lag.ms is not enforced accurately Key: KAFKA-14321 URL: https://issues.apache.org/jira/browse/KAFKA-14321 Project: Kafka Issue Type: Bug

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1308

See

[jira] [Resolved] (KAFKA-4545) tombstone needs to be removed after delete.retention.ms has passed after it has been cleaned

[ https://issues.apache.org/jira/browse/KAFKA-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-4545. Fix Version/s: 3.1.0 Resolution: Fixed This is fixed in KAFKA-8522. > tombstone needs to be

[jira] [Resolved] (KAFKA-14316) NoSuchElementException in feature control iterator

[ https://issues.apache.org/jira/browse/KAFKA-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-14316. - Fix Version/s: 3.4.0 3.3.2 Resolution: Fixed >

[jira] [Resolved] (KAFKA-13486) Kafka Connect: Failed to start task due to NPE

[ https://issues.apache.org/jira/browse/KAFKA-13486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Harris resolved KAFKA-13486. - Resolution: Duplicate > Kafka Connect: Failed to start task due to NPE >

[jira] [Created] (KAFKA-14320) Upgrade Jackson for CVE fixes

Javier Li Sam created KAFKA-14320: - Summary: Upgrade Jackson for CVE fixes Key: KAFKA-14320 URL: https://issues.apache.org/jira/browse/KAFKA-14320 Project: Kafka Issue Type: Bug

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1307

See Changes: -- [...truncated 427544 lines...] [2022-10-18T22:07:03.203Z] > Task :connect:api:javadocJar [2022-10-18T22:07:03.203Z] > Task :connect:api:compileTestJava

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi, David, Thanks for the reply. No more comments from me. 80. Yes, since PrepareAssignment returns topicIds, using topicId in ConsumerGroupInstallAssignmentRequest makes sense. 81. Sounds good. Jun On Tue, Oct 18, 2022 at 11:46 AM David Jacot wrote: > Hi Jun, > > 81. I forgot to say that I

Re: [DISCUSS] Apache Kafka 3.4.0 release

Hey all, I've created the release page for 3.4.0 with the current plan, which you can find here: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0 The freeze deadlines for this release are as follows: 1. KIP Freeze November 16th, 2022 2. Feature Freeze November 30th, 2022 3.

[jira] [Created] (KAFKA-14319) Storage tool format command does not work with old metadata versions

Jason Gustafson created KAFKA-14319: --- Summary: Storage tool format command does not work with old metadata versions Key: KAFKA-14319 URL: https://issues.apache.org/jira/browse/KAFKA-14319 Project:

[jira] [Created] (KAFKA-14318) KIP-878: Autoscaling for Statically Partitioned Streams

A. Sophie Blee-Goldman created KAFKA-14318: -- Summary: KIP-878: Autoscaling for Statically Partitioned Streams Key: KAFKA-14318 URL: https://issues.apache.org/jira/browse/KAFKA-14318 Project:

[jira] [Resolved] (KAFKA-14137) Security Vulnerabilities reported in CVE-2021-45046 and CVE-2021-45046

[ https://issues.apache.org/jira/browse/KAFKA-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Harris resolved KAFKA-14137. - Resolution: Fixed > Security Vulnerabilities reported in CVE-2021-45046 and CVE-2021-45046 >

[jira] [Created] (KAFKA-14317) Producer request timeouts are logged as network exceptions

Kirk True created KAFKA-14317: - Summary: Producer request timeouts are logged as network exceptions Key: KAFKA-14317 URL: https://issues.apache.org/jira/browse/KAFKA-14317 Project: Kafka Issue

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, 81. I forgot to say that I put UniformAssignor as the first one in the list. I think that it should be the default one. Best, David On Tue, Oct 18, 2022 at 8:33 PM David Jacot wrote: > > Hi Jun, > > 80. Hmm. It seems preferable to keep >

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, 80. Hmm. It seems preferable to keep ConsumerGroupInstallAssignmentRequest.Members.Partitions.TopicId as a topic id in order for the RPC to remain symmetrical with the PrepareAssignment RPCs. The client has to create the TopicPartitions from the mapping provided in the PrepareAssignment

[jira] [Created] (KAFKA-14316) NoSuchElementException in feature control iterator

Jason Gustafson created KAFKA-14316: --- Summary: NoSuchElementException in feature control iterator Key: KAFKA-14316 URL: https://issues.apache.org/jira/browse/KAFKA-14316 Project: Kafka

[jira] [Created] (KAFKA-14315) Kraft: 1 broker setup, broker took 34 seconds to transition from PrepareCommit to CompleteCommit

Travis Bischel created KAFKA-14315: -- Summary: Kraft: 1 broker setup, broker took 34 seconds to transition from PrepareCommit to CompleteCommit Key: KAFKA-14315 URL:

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi, David, Thanks for the reply. 80. The change in ConsumerGroupPrepareAssignmentResponse sounds good to me. Should ConsumerGroupInstallAssignmentRequest.Members.Partitions.TopicId be topic name? Intuitively, the client assignor assigns partitions based on topic names and it seems that the group

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

Thanks for all of the votes. I am going to go ahead and close the voting. 5 binding +1: Deng Ziming, David Jacot, David Arthur, Colin McCabe, Luke Chen -- -José

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Oops. I realized I just replied to Omnia 臘‍♀️ Here was my response for the mailing thread: Hey Omnia, Sorry to hear this is a problem for you as well. :( > * I have some concerns about the impact of this option on the transactional producers, for example, what will happen to an ongoing

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

Hi David, Thanks for the KIP, this is very exciting! How does JBOD relate to this work? KRaft mode doesn't yet support configuring brokers with multiple log directories. If the brokers in the existing cluster are configured with multiple log dirs, does the migration imply that the existing

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

Hi Colin, > I agree controlled shutdown is tricky to get just right. I suppose this is a > case where the RPCs we send out are not purely "fire and forget"; we have to > listen for the response. But that can be done in an event-based way. > Controlled shutdown is also probably the last thing

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1306

See Changes: -- [...truncated 427104 lines...] [2022-10-18T15:39:18.653Z] > Task :connect:api:testJar [2022-10-18T15:39:18.653Z] > Task :connect:api:testSrcJar

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1305

See

[jira] [Created] (KAFKA-14314) MirrorSourceConnector throwing NPE during `isCycle` check

John Krupka created KAFKA-14314: --- Summary: MirrorSourceConnector throwing NPE during `isCycle` check Key: KAFKA-14314 URL: https://issues.apache.org/jira/browse/KAFKA-14314 Project: Kafka

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, 80. I have included the mapping from topic ids to topic names in ConsumerGroupPrepareAssignmentResponse.Topics. It can be used to convert all the topic ids that you mentioned. It seems preferable to me to keep it like this as topic ids are usually smaller than topic names. Does that make

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Hi Omnia, Thanks for your reply. For (3), you said: > - I have some concerns about the impact of this option on the transactional producers, for example, what will happen to an ongoing transaction associated with an expired PID? Would this leave the transactions in a "hanging" state? - How will

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1304

See

Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

Hi Chris, 1. Thanks a lot for elaborating on this, I'm now convinced about the usefulness of the new offset reset endpoint. Regarding the follow-up KIP for a fine-grained offset write API, I'd be happy to take that on once this KIP is finalized and I will definitely look forward to your feedback

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

+1 from me. Thank you Luke On Fri, Oct 14, 2022 at 11:42 PM Colin McCabe wrote: > Thanks, José! +1 (binding) > > Colin > > > On Fri, Oct 14, 2022, at 07:54, David Arthur wrote: > > Thanks for the KIP, José! > > > > +1 (binding) > > > > -David > > > > On Fri, Oct 14, 2022 at 2:48 AM David Jacot