Ismael, I have updated the KIP to reuse the PolicyViolationException. It is not clear to me there is a value in either having a new exception type or in renaming the existing exception. For error codes, we'll just use POLICY_VIOLATION. This should work well for existing clients also since they have seen these exceptions and error codes from policies.
As you mention, the actual performance figures are probably better with trunk than for the version I used in the tests mentioned in the KIP. And as Colin mentions in another thread, the goal is to have Kafka support many more partitions in the future. I have tagged my performance experiments with a certain Kafka version so as to nor misrepresent Kafka's performance overall. Also, I think that the number of partitions is going to be a moving limit as faster and better Kafka versions come up. The experiments in the KIP were aimed at convincing the reader that we need some way to limit the number of partitions, rather than to specify a specific limit. Boyang, I have updated the KIP to reflect that until KIP-590, the limit won't apply to auto-created topics via Metadata API. After KIP-590, the Metadata API can also throw the PolicyViolationException (once it starts relying on CreateTopics). Do you think it makes sense to bump up the API versions if they are going to throw an exception type they could throw in the past? Or do we bump up the API version simply when there is a different reason why the API could throw a given exception? Thanks for your comments and feedback. On Mon, Jun 29, 2020 at 3:38 AM Boyang Chen <reluctanthero...@gmail.com> wrote: > Hey Gokul, thanks for the reply. > > It is true that the Metadata API will call CreateTopic under the cover. The > key guarantee we need to provide is to properly propagate the exact error > code to the original client. So either we are going to introduce a new > error code or reuse an existing one, the Metadata RPC should be able to > handle it. Could we mention this guarantee in the KIP, once the error code > discussion gets converged? > > On Wed, Jun 24, 2020 at 6:35 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > Error code names are safe to rename at the moment as they are in an > > internal package. The exception class is in a public package though. I > was > > thinking that PolicyViolationException could be a subclass of the more > > generic exception. This approach would mean that older clients would > > understand the error code, etc. I didn't think through all the details, > but > > worth considering. > > > > With regards to the version, OK. I expect trunk to do better than what > was > > described there on both replication overhead and topic creation/deletion > > performance. > > > > Ismael > > > > On Tue, Jun 23, 2020 at 11:10 PM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Ismael, > > > > > > I am open to using any error code and am not attached to one TBH. Colin > > had > > > suggested creating a new resource code called RESOURCE_LIMIT_EXCEEDED. > I > > am > > > happy to reuse the error code corresponding to PolicyViolation. Is it > > safe > > > to rename errors and corresponding exception names? If so, I'd prefer > > > reusing the existing code as well. > > > > > > For performance testing results that I added to this KIP, I used Kafka > > > 2.3.1, which was very close to trunk at the time I tested. We have seen > > > similar issues with Kafka 2.4.1. Please note that for the tests done in > > the > > > KIP, especially the Produce performance tests, we probably could have > > > gotten higher performance, but the focus was on comparing performance > > > across a different number of partitions, for a given configuration, > > rather > > > than trying to find out the best performance possible for the "right" > > > number of partitions. > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 24, 2020 at 4:07 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > Thanks for the KIP. A couple of questions: > > > > > > > > 1. Have we considered reusing the existing PolicyViolation error code > > and > > > > renaming it? This would make it simpler to handle on the client. > > > > > > > > 2. What version was used for the perf section? I think master should > do > > > > better than what's described there. > > > > > > > > Ismael > > > > > > > > On Wed, Apr 1, 2020, 8:28 AM Gokul Ramanan Subramanian < > > > > gokul24...@gmail.com> > > > > wrote: > > > > > > > > > Hi. > > > > > > > > > > I have opened KIP-578, intended to provide a mechanism to limit the > > > > number > > > > > of partitions in a Kafka cluster. Kindly provide feedback on the > KIP > > > > which > > > > > you can find at > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > > > > > > > > I want to specially thank Stanislav Kozlovski who helped in > > formulating > > > > > some aspects of the KIP. > > > > > > > > > > Many thanks, > > > > > > > > > > Gokul. > > > > > > > > > > > > > > >
