Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Ismael, I have updated the KIP to reuse the PolicyViolationException. It is not clear to me there is a value in either having a new exception type or in renaming the existing exception. For error codes, we'll just use POLICY_VIOLATION. This should work well for existing clients also since they

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hey Gokul, thanks for the reply. It is true that the Metadata API will call CreateTopic under the cover. The key guarantee we need to provide is to properly propagate the exact error code to the original client. So either we are going to introduce a new error code or reuse an existing one, the

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Error code names are safe to rename at the moment as they are in an internal package. The exception class is in a public package though. I was thinking that PolicyViolationException could be a subclass of the more generic exception. This approach would mean that older clients would understand the

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Alex, 904. For multi-tenancy, yes, the limits in this KIP will apply on a first-come-first-serve basis. If and when multi-tenancy based limiting rolls out, it would probably introduce a different way to specific per-client limits on partitions. I suppose those will apply in addition to and not

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Boyang, Tom and I discussed earlier in this thread whether we could use the topic creation policy for limiting the number of partitions. Basically, we realized that while we can use it for limiting the number of partitions, we cannot use it for assigning partitions in a way that respects the

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Ismael, I am open to using any error code and am not attached to one TBH. Colin had suggested creating a new resource code called RESOURCE_LIMIT_EXCEEDED. I am happy to reuse the error code corresponding to PolicyViolation. Is it safe to rename errors and corresponding exception names? If so, I'd

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Thanks for the KIP. A couple of questions: 1. Have we considered reusing the existing PolicyViolation error code and renaming it? This would make it simpler to handle on the client. 2. What version was used for the perf section? I think master should do better than what's described there.

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Gokul, Thanks for the excellent KIP. I was recently driving the rollout of KIP-590 and proposed to fix the hole of the bypassing of topic creation policy when applying

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Gokul, Thank you for the answers and the data provided to illustrate the use case. A couple of additional questions. 904. If multi-tenancy is addressed in a future KIP, how smooth would be the upgrade path? For example, the introduced configuration parameters still apply, right? We would

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Thanks Stanislav. Apologies about the long absence from this thread. I would prefer having per-user max partition limits in a separate KIP. I don't see this as an MVP for this KIP. I will add this as an alternative approach into the KIP. I was in a double mind about whether or not to impose the

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Thanks for the KIP, Gokul! I like the overall premise - I think it's more user-friendly to have configs for this than to have users implement their own config policy -> so unless it's very complex to implement, it seems worth it. I agree that having the topic policy on the CreatePartitions path

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Tom. With KIP-578, we are not trying to model the load on each partition, and come up with an exact limit on what the cluster or broker can handle in terms of number of partitions. We understand that not all partitions are equal, and the actual load per partition varies based on the message

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Gokul, the partition assignment algorithm needs to be aware of the partition > limits. > I agree, if you have limits then anything doing reassignment would need some way of knowing what they were. But the thing is that I'm not really sure how you would decide what the limits ought to be. >

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

This is good reference Tom. I did not consider this approach at all. I am happy to learn about it now. However, I think that partition limits are not "yet another" policy configuration. Instead, they are fundamental to partition assignment. i.e. the partition assignment algorithm needs to be

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Gokul, Leaving aside the question of how Kafka scales, I think the proposed solution, limiting the number of partitions in a cluster or per-broker, is a policy which ought to be addressable via the pluggable policies (e.g. create.topic.policy.class.name). Unfortunately although there's a

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi. For the sake of expediting the discussion, I have created a prototype PR: https://github.com/apache/kafka/pull/8499. Eventually, (if and) when the KIP is accepted, I'll modify this to add the full implementation and tests etc. in there. Would appreciate if a Kafka committer could share their

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Thanks for your comments Alex. The KIP proposes using two configurations max.partitions and max.broker.partitions. It does not enforce their use. The default values are pretty large (INT MAX), therefore should be non-intrusive. In multi-tenant environments and in partition assignment and

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi Gokul, Thanks for the KIP. >From what I understand, the objective of the new configuration is to protect a cluster from an overload driven by an excessive number of partitions independently from the load handled on the partitions themselves. As such, the approach uncouples the data-path load

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

Hi. Requesting you to take a look at this KIP and provide feedback. Thanks. Regards. On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > I have opened KIP-578, intended to provide a mechanism to limit the number > of partitions in a Kafka cluster.