For some reason, I have never gotten an email for this thread, but it seems like there was some activity and questions on this thread <https://lists.apache.org/thread/0vkwdpfwnzn6d4h6yc33b7zhwpcw64bz> while ago. I try to respond to the same email thread and hoping that this will work and my answers will add to the mailing list thread.
*> 1. The image you inserted doesn't get shown, please fix it* I am not sure if you are referring to the image under the motivation section of KIP-1010 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota>, or there is another image you are referring to. But clearly the one on confluence is showing up. Please clarify what you are not able to see. [image: Screenshot 2025-08-13 at 2.24.34 PM.png] *> 2. I'd like to clarify your proposal a bit. So for now we had (user, client), (user) or (client) combinations. You'd like to introduce topic-partitions in this framework. Would it extend the current behavior, so the previously 4 item set becomes a 6 item set like this: (tp, user, client), (tp, user), (tp, client), (tp), (user) or (client)? Or do these tp quotas behave differently?* We are following a similar pattern in this proposal, but we will not go as granular as 6 items. So, It will be (topic, partition) combination only, and it can co-exist with all the 3 combinations of user and client. The reason for limiting the granularity is a) it will be very difficult to configure, b) the metrics will have a very big number of dimensions (tags), c) we can configure both quotas and whichever happens first the client will be throttled. *> 3. How would your implementation work when the aggregate of topic quotas exceed the available bandwidth? Do topics get fair access or is it possible that some partitions can't be consumed because others eat the bandwidth?* I am not sure by "bandwidth", you mean the network bandwidth or the quota that is allocated to the topic. But, the implementation is similar to the user/client quota. Each topic-partition byte rate is reported separately, and if the consumption of that topic-partition reaches the quota, the next consumers will be throttled (same for producers). So, each partition has its quota and is separate from other partitions; however, if one partition gets throttled, all the requests that include that partition get throttled. *> 4. I'm a bit confused about the motivation section. So you're saying that if you have a topic with 6 partitions where a quota is set to 2MB/s, are you expecting a 4MB/s throughput if 2 of those topic's partition leaders are hosted on the broker? Wouldn't that violate backward compatibility because with a client now I can produce at 4MB/s rate? * There is no relation between the client quota and the partition byte rate with the current design. So, if you have a client quota of 4MB/s, the client can produce 4MB/s per broker. So, in a very extreme scenario, where all the leader partitions of the topic are on one broker, the client can produce 4MB/s for this topic. However, if they are evenly distributed, it can produce 4x6 MB/s. So, the motivation is to make the quota more predictable and make it independent from how the partitions are stacked. On Wed, Feb 14, 2024 at 7:04 AM Afshin Moazami <amoaz...@salesforce.com> wrote: > Thanks Viktor, > > Hi folks, > I would like to propose a new feature to extend the quota management in > Kafka to support topic-partition based quotas. The following is the link to > the KIP > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota > > > Best, > Afshin Moazami > > On Wed, Feb 7, 2024 at 5:25 AM Viktor Somogyi-Vass > <viktor.somo...@cloudera.com.invalid> wrote: > >> Hi Afshin, >> >> We keep KIP discussions on dev@kafka.apache.org so please post this over >> there too. I'll go over this later this week but devs usually monitor that >> list more frequently and you'll have better chances of getting a reply >> there. >> >> Regards, >> Viktor >> >> On Wed, Jan 17, 2024 at 12:03 AM Afshin Moazami >> <amoaz...@salesforce.com.invalid> wrote: >> >> > Hi folks, >> > I am not sure what is the KIP life-cycle and how we can get more >> attention >> > on them, so I just reply to this thread with the hope to get some >> > discussion started. >> > >> > Thanks, >> > Afshin >> > >> > On Mon, Dec 11, 2023 at 10:43 AM Afshin Moazami < >> amoaz...@salesforce.com> >> > wrote: >> > >> > > Hi folks, >> > > I would like to propose a new feature to extend the quota management >> in >> > > Kafka to support topic-partition based quotas. The following is the >> link >> > to >> > > the KIP >> > > >> > > >> > >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010*3A*Topic*Partition*Quota__;JSsrKw!!DCbAVzZNrAf4!BK-888ZjIeh53cmPcRZ_ZIpA6-02xIk5LXsT4cl82ieHRjWN31a-xsi36sN9I3P3LOhhpYCJU2FpbYkfg2YpGX2RXtPFAIjsHv0$ >> > > >> > > >> > > Best, >> > > Afshin Moazami >> > > >> > >> >
