Re: Problem with Kafka KRaft 3.1.X

Colin, hi, current max partitions reached is about 600,000 - I had to increase Linux file descriptors, mmap, and tweak the JVM heap settings a bit - heap error again. This is a bit of a hack to, as RF=1 and only a single EC2 instance - a proper 3 node cluster would in theory give >1M partitions

[jira] [Created] (KAFKA-14217) app-reset-tool.html should remove reference to --zookeeper flag that no longer exists

Colin McCabe created KAFKA-14217: Summary: app-reset-tool.html should remove reference to --zookeeper flag that no longer exists Key: KAFKA-14217 URL: https://issues.apache.org/jira/browse/KAFKA-14217

[jira] [Created] (KAFKA-14216) Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback javadoc

Colin McCabe created KAFKA-14216: Summary: Remove ZK reference from org.apache.kafka.server.quota.ClientQuotaCallback javadoc Key: KAFKA-14216 URL: https://issues.apache.org/jira/browse/KAFKA-14216

[jira] [Created] (KAFKA-14215) KRaft forwarded requests have no quota enforcement

Jason Gustafson created KAFKA-14215: --- Summary: KRaft forwarded requests have no quota enforcement Key: KAFKA-14215 URL: https://issues.apache.org/jira/browse/KAFKA-14215 Project: Kafka

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

Thanks Vicky, that reads much clearer now. Just regarding the value string name itself: "self.join" may be confusing compared to other values that people would think before this config is enabled, self-join are not allowed at all. Maybe we can rename it to "single.store.self.join"? Guozhang On

[jira] [Created] (KAFKA-14214) StandardAuthorizer may transiently process ACLs out of write order

Akhilesh Chaganti created KAFKA-14214: - Summary: StandardAuthorizer may transiently process ACLs out of write order Key: KAFKA-14214 URL: https://issues.apache.org/jira/browse/KAFKA-14214

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hello David, Alright I think that's sufficient. Just to make that clear in the doc, could we update: 1) the heartbeat request handling section, stating when coordinator will trigger rebalance based on the HB's member metadata / reason? 2) the "Rebalance Triggers" section to include what we

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #81

See Changes: -- [...truncated 303337 lines...] [2022-09-09T19:03:08.156Z] [2022-09-09T19:03:08.157Z] org.apache.kafka.streams.integration.ResetPartitionTimeIntegrationTest >

Re:[VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

+1 (non-binding) Really looking forward for the discussion on how other group types (especially connect) will support this new protocol. From: dev@kafka.apache.org At: 09/09/22 04:32:46 UTC-4:00To: dev@kafka.apache.org Subject: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

So it seems there's a consensus on having dedicated APIs for Connect, which means having data model (group, member, assignment) and APIs (heartbeat request/response, assignment prepare and install) tailored specifically to connect. I wonder if adding support for other coordinator group types

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #65

See Changes: -- [...truncated 573394 lines...] [2022-09-09T18:12:35.520Z] [2022-09-09T18:12:35.520Z] Exception: java.lang.AssertionError thrown from the

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1214

See Changes: -- [...truncated 510511 lines...] [2022-09-09T17:42:12.344Z] StreamsAssignmentScaleTest > testFallbackPriorTaskAssignorManyThreadsPerClient PASSED

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

Hello, Bill Please, cast your vote in VOTE thread - https://lists.apache.org/thread/hct5r7lz1n5mnwol56jhf5tlg2qykv5f > 9 сент. 2022 г., в 20:33, Bill Bejeck написал(а): > > Hi Николай, > > Thanks for the KIP; consistency

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

Hi Николай, Thanks for the KIP; consistency across the tooling is important, so I'm a +1 for this KIP. I agree with Guozhang on voting right away. -Bill On Fri, Sep 9, 2022 at 12:05 PM Guozhang Wang wrote: > Hi, I think we can vote right away. > > Guozhang > > On Fri, Sep 9, 2022 at 2:24 AM

Re: [DISCUSS] KIP-865 Metadata Transactions

Starting a new thread here https://lists.apache.org/thread/895pgb85l08g2l63k99cw5dt2qpjkxb9 On Fri, Sep 9, 2022 at 1:05 PM Colin McCabe wrote: > > Also, it looks like someone already claimed KIP-865, so I'd suggest grabbing > a new number. :) > > Colin > > > On Fri, Sep 9, 2022, at 09:38, Colin

[DISCUSS] KIP-868 Metadata Transactions (new thread)

Starting a new thread to avoid issues with mail client threading. Original thread follows: Hey folks, I'd like to start a discussion on the idea of adding transactions in the KRaft controller. This will allow us to overcome the current limitation of atomic batch sizes in Raft which lets us do

Re: [DISCUSS] KIP-865 Metadata Transactions

Also, it looks like someone already claimed KIP-865, so I'd suggest grabbing a new number. :) Colin On Fri, Sep 9, 2022, at 09:38, Colin McCabe wrote: > Thanks for this KIP, David! > > In the "motivation" section, it might help to give a concrete example > of an operation we want to be

Re: [VOTE] KIP-865: Support --bootstrap-server in kafka-streams-application-reset

+1. Thanks. Guozhang On Fri, Sep 9, 2022 at 9:52 AM Николай Ижиков wrote: > Hello. > > I'd like to start a vote on KIP-865 which adds support of > —bootstrap-server parameter in kafka-streams-application-reset tool > > Discuss Thread: >

Re: [DISCUSS] KIP-865 Metadata Transactions

Hello. Sorry, but it seems we should fix KIP numbering. KIP-865 [1] is another KIP :). [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-865%3A+Support+--bootstrap-server+in+kafka-streams-application-reset > 9 сент. 2022 г., в 19:38, Colin McCabe написал(а): > > Thanks for this KIP,

[VOTE] KIP-865: Support --bootstrap-server in kafka-streams-application-reset

Hello. I'd like to start a vote on KIP-865 which adds support of —bootstrap-server parameter in kafka-streams-application-reset tool Discuss Thread: https://lists.apache.org/thread/5c1plw7mgmzd4zzqh1w59cqopn8kv21c KIP:

Re: Problem with Kafka KRaft 3.1.X

Hi Paul, As Keith wrote, it does sound like you are hitting a separate Linux limit like the max mmap count. I'm curious how many partitions you can create if you change that config! best, Colin On Tue, Sep 6, 2022, at 14:02, Keith Paulson wrote: > I've had similar errors cause by mmap

Re: [DISCUSS] KIP-865 Metadata Transactions

Thanks for this KIP, David! In the "motivation" section, it might help to give a concrete example of an operation we want to be atomic. My favorite one is probably CreateTopics since it's easy to see that we want to create all of a topic or none of it, and a topic could be a potentially

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

Hi, I think we can vote right away. Guozhang On Fri, Sep 9, 2022 at 2:24 AM Николай Ижиков wrote: > Hello, Guozhang. > > Thanks for the feedback. > As this KIP very straightforward, is it worth to be voted right now? > Or should we wait more feedback? > > > 9 сент. 2022 г., в 08:11, Guozhang

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #133

See Changes: -- [...truncated 371399 lines...] [2022-09-09T14:55:06.255Z] > Task :clients:generatePomFileForMavenJavaPublication [2022-09-09T14:55:06.255Z]

[jira] [Created] (KAFKA-14213) Reduce lock contention between control-plane-kafka-request-handler and kafka-log-cleaner-thread

Lucas Wang created KAFKA-14213: -- Summary: Reduce lock contention between control-plane-kafka-request-handler and kafka-log-cleaner-thread Key: KAFKA-14213 URL: https://issues.apache.org/jira/browse/KAFKA-14213

[jira] [Created] (KAFKA-14212) Fetch error response when hitting public OAuth/OIDC provider

Sushant Mahajan created KAFKA-14212: --- Summary: Fetch error response when hitting public OAuth/OIDC provider Key: KAFKA-14212 URL: https://issues.apache.org/jira/browse/KAFKA-14212 Project: Kafka

Re: Transactions, delivery timeout and changing transactional producer behavior

Hi all, I would like to bump this and bring some attention to the issue. This is a nasty bug in the transactional producer, would be nice if I could get some feedback on the PR: https://github.com/apache/kafka/pull/12392 Thanks in advance, Daniel Viktor Somogyi-Vass ezt írta (időpont: 2022.

[DISCUSS] KIP-865 Metadata Transactions

Hey folks, I'd like to start a discussion on the idea of adding transactions in the KRaft controller. This will allow us to overcome the current limitation of atomic batch sizes in Raft which lets us do things like create topics with a huge number of partitions.

[jira] [Created] (KAFKA-14211) Streams log message has partition and offset transposed

Matt Allwood created KAFKA-14211: Summary: Streams log message has partition and offset transposed Key: KAFKA-14211 URL: https://issues.apache.org/jira/browse/KAFKA-14211 Project: Kafka

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #63

See Changes: -- [...truncated 574803 lines...] [2022-09-09T05:16:42.242Z] org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > testOuterLeft[caching enabled

Re: [DISCUSSION] KIP-864: Support --bootstrap-server in kafka-streams-application-reset

Hello, Guozhang. Thanks for the feedback. As this KIP very straightforward, is it worth to be voted right now? Or should we wait more feedback? > 9 сент. 2022 г., в 08:11, Guozhang Wang написал(а): > > Hello Николай, > > Thanks for writing the KIP, I think it's rather straightforward and

Re: [VOTE] KIP-862: Self-join optimization for stream-stream joins

Hey Guozhang, Ah it seems my text was not very clear :) With "TOPOLOGY_OPTIMIZATION_CONFIG will be extended to accept a list of optimization rule configs" I meant that it will accept the new value strings for each optimization rule. Let me rephrase that in the KIP to make it clearer. Is it better

[VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi all, Thank you all for the very positive discussion about KIP-848. It looks like folks are very positive about it overall. I would like to start a vote on KIP-848, which introduces a brand new consumer rebalance protocol. The KIP is here: https://cwiki.apache.org/confluence/x/HhD1D. Best,

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Guozhang, I thought that the assignor will always be consulted when the next heartbeat request is constructed. In other words, `PartitionAssignor#metadata` will be called for every heartbeat. This gives the opportunity for the assignor to enforce a rebalance by setting the reason to a non-zero

Re: [VOTE] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

Thanks everybody! This KIP has passed with 3 binding votes (Luke, David, Colin). Colin, I agree with the nitpick and have extended the KIP to mention throwing an informative error message. On Wed, Sep 7, 2022 at 11:09 PM Colin McCabe wrote: > +1 (binding). > > One nitpick: when the user sets

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

Thanks Ismael, I added an extra paragraph in the motivation. We have certainly hit this within our internal Confluent reassignment software and from a quick skim in the popular Cruise Control repository, I notice that similar problems have been hit there too. Hopefully the examples in the KIP are