Sounds great Boyang! Look forward to hearing back. Rangan
----- Original Message ----- From: Boyang Chen <[email protected]> To: RANGAN PRABHAKARAN, [email protected] At: 26-Feb-2020 13:45:01 Thanks Rangan for the input! We shall do a retro for Q1. I will reflect your use cases there and notify you if we have the next step plan to help out. Boyang ________________________________ From: Rangan Prabhakaran (BLOOMBERG/ 919 3RD A) <[email protected]> Sent: Tuesday, February 25, 2020 2:57 AM To: [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: Discussion thread for KIP-X Hi Boyang, The Kafka clusters we manage are multi-tenant clusters hosting anywhere from hundreds to a few thousand different workloads on any given cluster. For our setup, we have noticed that the breaking limit wrt partition count is around 10k partitions per broker. Beyond this point, we start seeing significant replication slowness, election slowness, issues around too many files opened etc The type of workloads on our clusters that would benefit from the proposal outlined in this KIP are * Bursty workloads such as workloads that flood the topic once an hour and need to be processed quickly within a strict time window * Workloads that are using topics as simple queues (stateless and don’t care about ordering within a partition) * Stream processing workloads where parallelism is driven by the number of input topic partitions Currently, we are over provisioning partitions to efficiently serve these workloads which results in significant under-utilization of the respective clusters. Additionally, we are also seeing quite a few workloads that are relying on the partition level ordering guarantees today and are filtering out the keys they don’t care about on the client side. These workloads would benefit from the key level ordering proposed in KIP-X and result in much simpler application logic for clients. Let me know if this helps and if you have any further questions Rangan From: [email protected] At: 02/21/20 15:45:49 To: [email protected]<mailto:[email protected]> Cc: Rangan Prabhakaran (BLOOMBERG/ 919 3RD A ) <mailto:[email protected]> , [email protected]<mailto:[email protected]> Subject: Re: Discussion thread for KIP-X Hey Rangan, thanks for the interest! In fact we are still in the design phase, and need more supporting use cases that requires a higher scaling factor than number of partitions. It would be good if you could share some of your needed use case when the unit time of processing one record is the bottleneck, or some cost wise concern of over-partitioning. Boyang On Fri, Feb 21, 2020 at 10:44 AM Guozhang Wang <[email protected]<mailto:[email protected]>> wrote: > cc @Boyang Chen <[email protected]<mailto:[email protected]>> who > authored this draft. > > > Guozhang > > On Fri, Feb 21, 2020 at 10:29 AM Rangan Prabhakaran (BLOOMBERG/ 919 3RD A) > < > [email protected]<mailto:[email protected]>> wrote: > > > Hi, > > A few of us have been following KIP-X. We are interested in the roadmap / > > plan there and would like to contribute towards the same. > > > > What are the next steps to discuss / iterate on this KIP ? Currently, its > > in draft state and there does not seem to be a discussion thread attached > > yet. > > > > KIP - > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-X%3A+Introduce+a+cooperati ve+consumer+processing+semantic<https://cwiki.apache.org/confluence/display/KAFKA/KIP-X%3A+Introduce+a+cooperative+consumer+processing+semantic> > > > > Thanks > > Rangan > > > > -- > -- Guozhang >
