Re: Improving crash-restart catchup speed
Thanks. Makes sense. On Wed, Dec 16, 2020 at 2:02 AM Haruki Okada wrote: > Hi. > One possible solution I can imagine is to use replication throttle. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas > > You can set lower replication quota for other brokers -> failed broker > traffic during catching-up to prevent exhausting storage throughput. > > 2020年12月15日(火) 20:56 Gokul Ramanan Subramanian : > > > Hi. > > > > > > When a broker crashes and restarts after a long time (say 30 minutes), > > the broker has to do some work to sync all its replicas with the > > leaders. If the broker is a preferred replica for some partition, the > > broker may become the leader as part of a preferred replica leader > > election, while it is still catching up on some other partitions. > > > > > > This scenario can lead to a high incoming throughput on the broker > > during the catch up phase and cause back pressure with certain storage > > volumes (which have a fixed max throughput). This backpressure can > > slow down recovery time, and manifest in the form of client > > application errors in producing / consuming data on / from the > > recovering broker. > > > > > > I am looking for solutions to mitigate this problem. There are two > > solutions that I am aware of. > > > > > > 1. Scale out the cluster to have more brokers, so that the replication > > traffic is smaller per broker during recovery. > > > > > > 2. Keep preferred replica leader elections disabled and manually run > > one instance of preferred replica leader election after the broker has > > come back up and fully caught up. > > > > > > Are there other solutions? > > > > > -- > > Okada Haruki > ocadar...@gmail.com > >
Improving crash-restart catchup speed
Hi. When a broker crashes and restarts after a long time (say 30 minutes), the broker has to do some work to sync all its replicas with the leaders. If the broker is a preferred replica for some partition, the broker may become the leader as part of a preferred replica leader election, while it is still catching up on some other partitions. This scenario can lead to a high incoming throughput on the broker during the catch up phase and cause back pressure with certain storage volumes (which have a fixed max throughput). This backpressure can slow down recovery time, and manifest in the form of client application errors in producing / consuming data on / from the recovering broker. I am looking for solutions to mitigate this problem. There are two solutions that I am aware of. 1. Scale out the cluster to have more brokers, so that the replication traffic is smaller per broker during recovery. 2. Keep preferred replica leader elections disabled and manually run one instance of preferred replica leader election after the broker has come back up and fully caught up. Are there other solutions?
Re: Why should SASL principal be unchanged upon reauth
Thanks Ron. Makes sense. I'll pitch the idea of writing this KIP to my team. On Mon, Sep 28, 2020 at 11:01 PM Ron Dagostino wrote: > Hi Gokul. This would require a KIP to discuss. If you have a compelling > need to change this then I encourage you to create a KIP and start the > discussion. I’m not aware of anything specific that would prevent the > change, but I will say that my gut instinct says it wouldn’t be a good idea > without a truly compelling justification. > > Ron > > > On Sep 28, 2020, at 8:34 AM, Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > > Thanks Ron. > > > > Does this mean that it would be safe to remove the "KafkaPrincipal has > not > > changed after reauth" check in a future version? Or do you think there > are > > portions of the code that work under the assumption that this check > exists? > > > >> On Sun, Sep 27, 2020 at 12:07 AM Ron Dagostino > wrote: > >> > >> Hi Gokul. I looked back at the discussion thread, and it appears it > was an > >> arbitrary decision. > >> > >> > >> > https://lists.apache.org/thread.html/45c09f226386c0b1dc5f9b36e112882a20414d5900f8d778969e633e%40%3Cdev.kafka.apache.org%3E > >> > >> Ron > >> > >> On Thu, Sep 24, 2020 at 11:03 AM Gokul Ramanan Subramanian < > >> gokul24...@gmail.com> wrote: > >> > >>> Hi. > >>> > >>> I was looking through Kafka code and found that SASL KafkaPrincipals > are > >>> not supposed to change upon reauthentication, and if they do, the > broker > >>> will kill the TCP connection. > >>> > >>> What is the reasoning behind this limitation? > >>> > >>> Thanks. > >>> > >> >
Re: Why should SASL principal be unchanged upon reauth
Thanks Ron. Does this mean that it would be safe to remove the "KafkaPrincipal has not changed after reauth" check in a future version? Or do you think there are portions of the code that work under the assumption that this check exists? On Sun, Sep 27, 2020 at 12:07 AM Ron Dagostino wrote: > Hi Gokul. I looked back at the discussion thread, and it appears it was an > arbitrary decision. > > > https://lists.apache.org/thread.html/45c09f226386c0b1dc5f9b36e112882a20414d5900f8d778969e633e%40%3Cdev.kafka.apache.org%3E > > Ron > > On Thu, Sep 24, 2020 at 11:03 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi. > > > > I was looking through Kafka code and found that SASL KafkaPrincipals are > > not supposed to change upon reauthentication, and if they do, the broker > > will kill the TCP connection. > > > > What is the reasoning behind this limitation? > > > > Thanks. > > >
Why should SASL principal be unchanged upon reauth
Hi. I was looking through Kafka code and found that SASL KafkaPrincipals are not supposed to change upon reauthentication, and if they do, the broker will kill the TCP connection. What is the reasoning behind this limitation? Thanks.
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi, at this point, this KIP has one binding vote from Harsha. Does anyone else have an opinion on this KIP? Do you think we should go forward with this KIP once we release Kafka 3.0, since there wouldn't be the ZK backdoor in 3.0, making the partition limits properly enforceable? Thanks. On Wed, Aug 5, 2020 at 5:59 PM Harsha Ch wrote: > Thanks for the updates Gokul. +1 (binding). > > Thanks, > > Harsha > > On Wed, Aug 05, 2020 at 8:18 AM, Gokul Ramanan Subramanian < > gokul24...@gmail.com > wrote: > > > > > > > > > Hi. > > > > > > > > I request the community to kindly resume the voting process for KIP-578. > > If you have further comments, we can address those as well. > > > > > > > > Thanks and cheers. > > > > > > > > On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul2411s@ > gmail. > > com ( gokul24...@gmail.com ) > wrote: > > > > > >> > >> > >> Hi. > >> > >> > >> > >> Can we resume the voting process for KIP-578? I have addressed > additional > >> comments by Boyang and Ismael. > >> > >> > >> > >> Thanks. > >> > >> > >> > >> On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul2411s@ > gmail. > >> com ( gokul24...@gmail.com ) > wrote: > >> > >> > >>> > >>> > >>> Hi. Can we resume the voting process for KIP-578? Thanks. > >>> > >>> > >>> > >>> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul2411s@ > gmail. > >>> com ( gokul24...@gmail.com ) > wrote: > >>> > >>> > >>>> > >>>> > >>>> Thanks Colin. Have updated the KIP per your recommendations. Let me > know > >>>> what you think. > >>>> > >>>> > >>>> > >>>> Thanks Harsha for the vote. > >>>> > >>>> > >>>> > >>>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe < cmccabe@ apache. org ( > >>>> cmcc...@apache.org ) > wrote: > >>>> > >>>> > >>>>> > >>>>> > >>>>> Hi Gokul Ramanan Subramanian, > >>>>> > >>>>> > >>>>> > >>>>> Thanks for the KIP. > >>>>> > >>>>> > >>>>> > >>>>> Can you please modify the KIP to remove the reference to the > deprecated > >>>>> --zookeeper flag? This is not how kafka-configs. sh ( > >>>>> http://kafka-configs.sh/ ) is supposed to be used in new versions > of Kafka. > >>>>> You get a warning message if you do use this deprecated flag. As > described > >>>>> in KIP-604, we are removing the --zookeeper flag in the Kafka 3.0 > release. > >>>>> It also causes problems when people use the deprecated access mode-- > for > >>>>> example, as you note in this KIP, it bypasses resource limits such > as the > >>>>> ones described here. > >>>>> > >>>>> > >>>>> > >>>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about > RESOURCE_LIMIT_REACHED? > >>>>> Then the error string can contain the detailed message about which > >>>>> resource limit was hit (per broker limit, per cluster limit, > whatever.) It > >>>>> would also be good to spell out that CreateTopicsPolicy plugins can > also > >>>>> throw this exception, for consistency. > >>>>> > >>>>> > >>>>> > >>>>> I realize that 2 billion partitions seems like a very big number. > However, > >>>>> filesystems have had to transition to 64 bit inode numbers as time > has > >>>>> gone on. There doesn't seem to be any performance reason why this > should > >>>>> be a 31 bit number, so let's just make these configurations longs, > not > >>>>> ints. > >>>>> > >>>>> > >>>>> > >>>>> best, > >>>>> Colin > >>>>> > >>>>> > >>>>> > >>>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: > >>>&
Re: [DISCUSS] KIP-516: Topic Identifiers
Hi. Thanks for the KIP. Have you thought about whether it makes sense to support authorizing a principal for a topic ID rather than a topic name to achieve tighter security? Or is the topic ID fundamentally an internal detail similar to epochs used in a bunch of other places in Kafka? Thanks. On Fri, Sep 11, 2020 at 4:06 PM John Roesler wrote: > Hello Justine, > > Thanks for the KIP! > > I happen to have been confronted recently with the need to keep track of a > large number of topics as compactly as possible. I was going to come up > with some way to dictionary encode the topic names as integers, but this > seems much better! > > Apologies if this has been raised before, but I’m wondering about the > choice of UUID vs sequence number for the ids. Typically, I’ve seen UUIDs > in two situations: > 1. When processes need to generate non-colliding identifiers without > coordination. > 2. When the identifier needs to be “universally unique”; I.e., the > identifier needs to distinguish the entity from all other entities that > could ever exist. This is useful in cases where entities from all kinds of > systems get mixed together, such as when dumping logs from all processes in > a company into a common system. > > Maybe I’m being short-sighted, but it doesn’t seem like either really > applies here. It seems like the brokers could and would achieve consensus > when creating a topic anyway, which is all that’s required to generate > non-colliding sequence ids. For the second, as you mention, we could assign > a UUID to the cluster as a whole, which would render any resource scoped to > the broker universally unique as well. > > The reason I mention this is that, although a UUID is way more compact > than topic names, it’s still 16 bytes. In contrast, a 4-byte integer > sequence id would give us 4 billion unique topics per cluster, which seems > like enough ;) > > Considering the number of different times these topic identifiers are sent > over the wire or stored in memory, it seems like it might be worth the > additional 4x space savings. > > What do you think about this? > > Thanks, > John > > On Fri, Sep 11, 2020, at 03:20, Tom Bentley wrote: > > Hi Justine, > > > > This looks like a very welcome improvement. Thanks! > > > > Maybe I missed it, but the KIP doesn't seem to mention changing > > DeleteTopicsRequest to identify the topic using an id. Maybe that's out > of > > scope, but DeleteTopicsRequest is not listed among the Future Work APIs > > either. > > > > Kind regards, > > > > Tom > > > > On Thu, Sep 10, 2020 at 3:59 PM Satish Duggana > > > wrote: > > > > > Thanks Lucas/Justine for the nice KIP. > > > > > > It has several benefits which also include simplifying the topic > > > deletion process by controller and logs cleanup by brokers in corner > > > cases. > > > > > > Best, > > > Satish. > > > > > > On Wed, Sep 9, 2020 at 10:07 PM Justine Olshan > > > wrote: > > > > > > > > Hello all, it's been almost a year! I've made some changes to this > KIP > > > and hope to continue the discussion. > > > > > > > > One of the main changes I've added is now the metadata response will > > > include the topic ID (as Colin suggested). Clients can obtain the > topicID > > > of a given topic through a TopicDescription. The topicId will also be > > > included with the UpdateMetadata request. > > > > > > > > Let me know what you all think. > > > > Thank you, > > > > Justine > > > > > > > > On 2019/09/13 16:38:26, "Colin McCabe" wrote: > > > > > Hi Lucas, > > > > > > > > > > Thanks for tackling this. Topic IDs are a great idea, and this is > a > > > really good writeup. > > > > > > > > > > For /brokers/topics/[topic], the schema version should be bumped to > > > version 3, rather than 2. KIP-455 bumped the version of this znode to > 2 > > > already :) > > > > > > > > > > Given that we're going to be seeing these things as strings as lot > (in > > > logs, in ZooKeeper, on the command-line, etc.), does it make sense to > use > > > base64 when converting them to strings? > > > > > > > > > > Here is an example of the hex representation: > > > > > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8 > > > > > > > > > > And here is an example in base64. > > > > > b8tRS7h4TJ2Vt43Dp85v2A > > > > > > > > > > The base64 version saves 15 letters (to be fair, 4 of those were > > > dashes that we could have elided in the hex representation.) > > > > > > > > > > Another thing to consider is that we should specify that the > > > all-zeroes UUID is not a valid topic UUID. We can't use null for this > > > because we can't pass a null UUID over the RPC protocol (there is no > > > special pattern for null, nor do we want to waste space reserving such > a > > > pattern.) > > > > > > > > > > Maybe I missed it, but did you describe "migration of... existing > > > topic[s] without topic IDs" in detail in any section? It seems like > when > > > the new controller becomes active, it should just generate random > UUIDs for > > > these, and write the
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi. I request the community to kindly resume the voting process for KIP-578. If you have further comments, we can address those as well. Thanks and cheers. On Tue, Jul 21, 2020 at 2:07 PM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > Can we resume the voting process for KIP-578? I have addressed additional > comments by Boyang and Ismael. > > Thanks. > > On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > >> Hi. Can we resume the voting process for KIP-578? Thanks. >> >> On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < >> gokul24...@gmail.com> wrote: >> >>> Thanks Colin. Have updated the KIP per your recommendations. Let me know >>> what you think. >>> >>> Thanks Harsha for the vote. >>> >>> On Wed, May 27, 2020 at 8:17 PM Colin McCabe wrote: >>> >>>> Hi Gokul Ramanan Subramanian, >>>> >>>> Thanks for the KIP. >>>> >>>> Can you please modify the KIP to remove the reference to the deprecated >>>> --zookeeper flag? This is not how kafka-configs.sh is supposed to be used >>>> in new versions of Kafka. You get a warning message if you do use this >>>> deprecated flag. As described in KIP-604, we are removing the --zookeeper >>>> flag in the Kafka 3.0 release. It also causes problems when people use the >>>> deprecated access mode-- for example, as you note in this KIP, it bypasses >>>> resource limits such as the ones described here. >>>> >>>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about >>>> RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed >>>> message about which resource limit was hit (per broker limit, per cluster >>>> limit, whatever.) It would also be good to spell out that >>>> CreateTopicsPolicy plugins can also throw this exception, for consistency. >>>> >>>> I realize that 2 billion partitions seems like a very big number. >>>> However, filesystems have had to transition to 64 bit inode numbers as time >>>> has gone on. There doesn't seem to be any performance reason why this >>>> should be a 31 bit number, so let's just make these configurations longs, >>>> not ints. >>>> >>>> best, >>>> Colin >>>> >>>> >>>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: >>>> > Thanks for the KIP Gokul. This will be really useful for our use >>>> cases as >>>> > well. >>>> > +1 (binding). >>>> > >>>> > -Harsha >>>> > >>>> > >>>> > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < >>>> > gokul24...@gmail.com> wrote: >>>> > >>>> > > Hi. >>>> > > >>>> > > Any votes for this? >>>> > > >>>> > > Thanks. >>>> > > >>>> > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < >>>> gokul2411s@ >>>> > > gmail.com> wrote: >>>> > > >>>> > > Hello, >>>> > > >>>> > > I'd like to initialize voting on KIP-578: >>>> > > https://cwiki.apache.org/confluence/display/KAFKA/ >>>> > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions >>>> > > . >>>> > > >>>> > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez >>>> and Tom >>>> > > Bentley on the discussion thread. I have addressed their comments. >>>> I want >>>> > > to thank them for their time. >>>> > > >>>> > > If there are any more concerns about the KIP, I am happy to discuss >>>> them >>>> > > further. >>>> > > >>>> > > Thanks. >>>> > > >>>> > > >>>> > >>>> >>>
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi. Can we resume the voting process for KIP-578? I have addressed additional comments by Boyang and Ismael. Thanks. On Mon, Jun 8, 2020 at 9:09 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. Can we resume the voting process for KIP-578? Thanks. > > On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > >> Thanks Colin. Have updated the KIP per your recommendations. Let me know >> what you think. >> >> Thanks Harsha for the vote. >> >> On Wed, May 27, 2020 at 8:17 PM Colin McCabe wrote: >> >>> Hi Gokul Ramanan Subramanian, >>> >>> Thanks for the KIP. >>> >>> Can you please modify the KIP to remove the reference to the deprecated >>> --zookeeper flag? This is not how kafka-configs.sh is supposed to be used >>> in new versions of Kafka. You get a warning message if you do use this >>> deprecated flag. As described in KIP-604, we are removing the --zookeeper >>> flag in the Kafka 3.0 release. It also causes problems when people use the >>> deprecated access mode-- for example, as you note in this KIP, it bypasses >>> resource limits such as the ones described here. >>> >>> Instead of WILL_EXCEED_PARTITION_LIMITS, how about >>> RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed >>> message about which resource limit was hit (per broker limit, per cluster >>> limit, whatever.) It would also be good to spell out that >>> CreateTopicsPolicy plugins can also throw this exception, for consistency. >>> >>> I realize that 2 billion partitions seems like a very big number. >>> However, filesystems have had to transition to 64 bit inode numbers as time >>> has gone on. There doesn't seem to be any performance reason why this >>> should be a 31 bit number, so let's just make these configurations longs, >>> not ints. >>> >>> best, >>> Colin >>> >>> >>> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: >>> > Thanks for the KIP Gokul. This will be really useful for our use cases >>> as >>> > well. >>> > +1 (binding). >>> > >>> > -Harsha >>> > >>> > >>> > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < >>> > gokul24...@gmail.com> wrote: >>> > >>> > > Hi. >>> > > >>> > > Any votes for this? >>> > > >>> > > Thanks. >>> > > >>> > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < >>> gokul2411s@ >>> > > gmail.com> wrote: >>> > > >>> > > Hello, >>> > > >>> > > I'd like to initialize voting on KIP-578: >>> > > https://cwiki.apache.org/confluence/display/KAFKA/ >>> > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions >>> > > . >>> > > >>> > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez >>> and Tom >>> > > Bentley on the discussion thread. I have addressed their comments. I >>> want >>> > > to thank them for their time. >>> > > >>> > > If there are any more concerns about the KIP, I am happy to discuss >>> them >>> > > further. >>> > > >>> > > Thanks. >>> > > >>> > > >>> > >>> >>
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Ismael, I have updated the KIP to reuse the PolicyViolationException. It is not clear to me there is a value in either having a new exception type or in renaming the existing exception. For error codes, we'll just use POLICY_VIOLATION. This should work well for existing clients also since they have seen these exceptions and error codes from policies. As you mention, the actual performance figures are probably better with trunk than for the version I used in the tests mentioned in the KIP. And as Colin mentions in another thread, the goal is to have Kafka support many more partitions in the future. I have tagged my performance experiments with a certain Kafka version so as to nor misrepresent Kafka's performance overall. Also, I think that the number of partitions is going to be a moving limit as faster and better Kafka versions come up. The experiments in the KIP were aimed at convincing the reader that we need some way to limit the number of partitions, rather than to specify a specific limit. Boyang, I have updated the KIP to reflect that until KIP-590, the limit won't apply to auto-created topics via Metadata API. After KIP-590, the Metadata API can also throw the PolicyViolationException (once it starts relying on CreateTopics). Do you think it makes sense to bump up the API versions if they are going to throw an exception type they could throw in the past? Or do we bump up the API version simply when there is a different reason why the API could throw a given exception? Thanks for your comments and feedback. On Mon, Jun 29, 2020 at 3:38 AM Boyang Chen wrote: > Hey Gokul, thanks for the reply. > > It is true that the Metadata API will call CreateTopic under the cover. The > key guarantee we need to provide is to properly propagate the exact error > code to the original client. So either we are going to introduce a new > error code or reuse an existing one, the Metadata RPC should be able to > handle it. Could we mention this guarantee in the KIP, once the error code > discussion gets converged? > > On Wed, Jun 24, 2020 at 6:35 AM Ismael Juma wrote: > > > Error code names are safe to rename at the moment as they are in an > > internal package. The exception class is in a public package though. I > was > > thinking that PolicyViolationException could be a subclass of the more > > generic exception. This approach would mean that older clients would > > understand the error code, etc. I didn't think through all the details, > but > > worth considering. > > > > With regards to the version, OK. I expect trunk to do better than what > was > > described there on both replication overhead and topic creation/deletion > > performance. > > > > Ismael > > > > On Tue, Jun 23, 2020 at 11:10 PM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Ismael, > > > > > > I am open to using any error code and am not attached to one TBH. Colin > > had > > > suggested creating a new resource code called RESOURCE_LIMIT_EXCEEDED. > I > > am > > > happy to reuse the error code corresponding to PolicyViolation. Is it > > safe > > > to rename errors and corresponding exception names? If so, I'd prefer > > > reusing the existing code as well. > > > > > > For performance testing results that I added to this KIP, I used Kafka > > > 2.3.1, which was very close to trunk at the time I tested. We have seen > > > similar issues with Kafka 2.4.1. Please note that for the tests done in > > the > > > KIP, especially the Produce performance tests, we probably could have > > > gotten higher performance, but the focus was on comparing performance > > > across a different number of partitions, for a given configuration, > > rather > > > than trying to find out the best performance possible for the "right" > > > number of partitions. > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 24, 2020 at 4:07 AM Ismael Juma wrote: > > > > > > > Thanks for the KIP. A couple of questions: > > > > > > > > 1. Have we considered reusing the existing PolicyViolation error code > > and > > > > renaming it? This would make it simpler to handle on the client. > > > > > > > > 2. What version was used for the perf section? I think master should > do > > > > better than what's described there. > > > > > > > > Ismael > > > > > > > > On Wed, Apr 1, 2020, 8:28 AM Gokul Ramanan Subramanian < > >
Re: Running system tests on mac
Thanks Colin.
While at the subject of system tests, there are a few times I see tests
timed out (even on a large machine such as m5.4xlarge EC2 with Linux). Are
there any knobs that system tests provide to control timeouts / throughputs
across all tests?
Thanks.
On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe wrote:
> Ducktape runs on Python 2. You can't use it with Python 3, as you are
> trying to do here.
>
> If anyone's interested in porting it to Python 3 it would be a good change.
>
> Otherwise, using docker as suggested here seems to be the best way to go.
>
> best,
> Colin
>
> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
> > Hi.
> >
> > Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
> > Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
> > *ducktape tests/kafkatest/tests* yields the following error, making it
> look
> > like some Python incompatibility issue.
> >
> > $ ducktape tests/kafkatest/tests
> > Traceback (most recent call last):
> > File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
> >
> > load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 487, in load_entry_point
> > return get_distribution(dist).load_entry_point(group, name)
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2728, in load_entry_point
> > return ep.load()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2346, in load
> > return self.resolve()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2352, in resolve
> > module = __import__(self.module_name, fromlist=['__name__'],
> > level=0)
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> > line 127
> > print "parameters are not valid json: " + str(e.message)
> > ^
> > SyntaxError: invalid syntax
> >
> > I followed the instructions in tests/README.md to setup a cluster of 9
> > worker machines. That worked well. When I ran *python setup.py develop*
> to
> > install the necessary dependencies (including ducktape), I got similar
> > errors to above, but the overall command completed successfully.
> >
> > Any help appreciated.
> >
> > Thanks.
> >
>
Re: Running system tests on mac
I have been running the tests for over 5 hours now, and only 60 or so out
of the 600 tests are through. Many tests are timing out. For example:
[INFO:2020-06-29 07:52:54,606]: RunnerClient:
kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test.StreamsCooperativeRebalanceUpgradeTest.test_upgrade_to_cooperative_rebalance.upgrade_from_version=2.3.1:
FAIL: Never saw 'Processed [0-9]* records so far' message ducker@ducker07
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py",
line 132, in run
data = self.run_test()
File
"/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py",
line 189, in run_test
return self.test_context.function(self.test)
File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py",
line 428, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py",
line 90, in test_upgrade_to_cooperative_rebalance
verify_running(processor, self.processing_message)
File "/opt/kafka-dev/tests/kafkatest/tests/streams/utils/util.py", line
21, in verify_running
err_msg="Never saw '%s' message " % message +
str(processor.node.account))
File
"/usr/local/lib/python2.7/dist-packages/ducktape/cluster/remoteaccount.py",
line 705, in wait_until
allow_fail=True) == 0, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ducktape/utils/util.py",
line 41, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg)
TimeoutError: Never saw 'Processed [0-9]* records so far' message
ducker@ducker07
My Macbook pro has over 4GB of RAM free and between 15-45 % idle CPU. So, I
don't think I am running low on resources.
Is this common? At this rate, the tests would take about 2 days to complete
and there'd probably be lots of failures.
On Mon, Jun 29, 2020 at 11:21 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:
> Thanks. This worked. I had to use the Docker desktop app, not
> docker-machine, with which it gave errors around permissions setting up
> /etc/hosts.
>
> On Mon, Jun 29, 2020 at 10:24 AM Nikolay Izhikov
> wrote:
>
>> Hello,
>>
>> I successfully run system tests on Mac with Docker.
>> I followed the instruction on [1] and it works like a charm.
>>
>> [1]
>> https://github.com/apache/kafka/tree/trunk/tests#running-tests-using-docker
>>
>>
>> > 29 июня 2020 г., в 12:14, Gokul Ramanan Subramanian <
>> gokul24...@gmail.com> написал(а):
>> >
>> > Hi.
>> >
>> > Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
>> > Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
>> > *ducktape tests/kafkatest/tests* yields the following error, making it
>> look
>> > like some Python incompatibility issue.
>> >
>> > $ ducktape tests/kafkatest/tests
>> > Traceback (most recent call last):
>> > File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
>> >
>> >load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>> > File
>> >
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>> > line 487, in load_entry_point
>> >return get_distribution(dist).load_entry_point(group, name)
>> > File
>> >
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>> > line 2728, in load_entry_point
>> >return ep.load()
>> > File
>> >
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>> > line 2346, in load
>> >return self.resolve()
>> > File
>> >
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>> > line 2352, in resolve
>> >module = __import__(self.module_name, fromlist=['__name__'], level=0)
>> > File
>> >
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>> > line 127
>> >print "parameters are not valid json: " + str(e.message)
>> > ^
>> > SyntaxError: invalid syntax
>> >
>> > I followed the instructions in tests/README.md to setup a cluster of 9
>> > worker machines. That worked well. When I ran *python setup.py develop*
>> to
>> > install the necessary dependencies (including ducktape), I got similar
>> > errors to above, but the overall command completed successfully.
>> >
>> > Any help appreciated.
>> >
>> > Thanks.
>>
>>
Re: Running system tests on mac
Thanks. This worked. I had to use the Docker desktop app, not
docker-machine, with which it gave errors around permissions setting up
/etc/hosts.
On Mon, Jun 29, 2020 at 10:24 AM Nikolay Izhikov
wrote:
> Hello,
>
> I successfully run system tests on Mac with Docker.
> I followed the instruction on [1] and it works like a charm.
>
> [1]
> https://github.com/apache/kafka/tree/trunk/tests#running-tests-using-docker
>
>
> > 29 июня 2020 г., в 12:14, Gokul Ramanan Subramanian <
> gokul24...@gmail.com> написал(а):
> >
> > Hi.
> >
> > Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
> > Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
> > *ducktape tests/kafkatest/tests* yields the following error, making it
> look
> > like some Python incompatibility issue.
> >
> > $ ducktape tests/kafkatest/tests
> > Traceback (most recent call last):
> > File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
> >
> >load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 487, in load_entry_point
> >return get_distribution(dist).load_entry_point(group, name)
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2728, in load_entry_point
> >return ep.load()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2346, in load
> >return self.resolve()
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 2352, in resolve
> >module = __import__(self.module_name, fromlist=['__name__'], level=0)
> > File
> >
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> > line 127
> >print "parameters are not valid json: " + str(e.message)
> > ^
> > SyntaxError: invalid syntax
> >
> > I followed the instructions in tests/README.md to setup a cluster of 9
> > worker machines. That worked well. When I ran *python setup.py develop*
> to
> > install the necessary dependencies (including ducktape), I got similar
> > errors to above, but the overall command completed successfully.
> >
> > Any help appreciated.
> >
> > Thanks.
>
>
Running system tests on mac
Hi.
Has anyone had luck running Kafka system tests on a Mac. I have a MacOS
Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
*ducktape tests/kafkatest/tests* yields the following error, making it look
like some Python incompatibility issue.
$ ducktape tests/kafkatest/tests
Traceback (most recent call last):
File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11, in
load_entry_point('ducktape', 'console_scripts', 'ducktape')()
File
"/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 487, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File
"/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 2728, in load_entry_point
return ep.load()
File
"/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 2346, in load
return self.resolve()
File
"/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
line 2352, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File
"/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
line 127
print "parameters are not valid json: " + str(e.message)
^
SyntaxError: invalid syntax
I followed the instructions in tests/README.md to setup a cluster of 9
worker machines. That worked well. When I ran *python setup.py develop* to
install the necessary dependencies (including ducktape), I got similar
errors to above, but the overall command completed successfully.
Any help appreciated.
Thanks.
Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2
Thanks Gwen and Ismael. On Wed, Jun 24, 2020 at 6:52 PM Gwen Shapira wrote: > Kafka follows a "fully compatible model" and new features are always > optional. We have extensive compatibility testing to make sure rolling > upgrades are safe and painless. > This investment in compatibility allows us to reduce the effort needed in > maintaining old versions (Security CVE is the obvious exception, as well as > data-loss bugs). > > Not every OSS project has the same model, but it has worked for Kafka for > years now (since the awful upgrade from 0.7 to 0.8 in 2014). > > Gwen > > > On Wed, Jun 24, 2020 at 10:40 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > I agree that is an option, but is there any reason to not have a 2.3.2 > and > > 2.4.2 released? If so, it would be nice to know about these reasons. > > Appreciate your time on this. > > > > On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma wrote: > > > > > I think there's some confusion here. You can upgrade to AK 2.5.0 and > > > completely ignore ZK and TLS. It's completely optional. > > > > > > Ismael > > > > > > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian < > > > gokul24...@gmail.com> wrote: > > > > > > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of > migrating > > > > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it > > > would > > > > be nice to have a bug-free version of 2.3 and 2.4. > > > > > > > > Is there a technical reason why we hesitate to get these versions > out? > > Or > > > > is it that no one has got around to it? > > > > > > > > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia < > > > sankalpbhati...@gmail.com> > > > > wrote: > > > > > > > > > Thanks Ismael for the response. > > > > > > > > > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which > > had > > > > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it > > > > > introduces some major changes like support for ZK encryption and a > > few > > > > > others, I was wondering if we should choose a smaller upgrade in > such > > > > cases > > > > > as we don't really require the new features in 2.5 and above right > > now. > > > > > > > > > > - > > > > > Sankalp > > > > > > > > > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma > wrote: > > > > > > > > > > > Hi Sankalp, > > > > > > > > > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 > > > instead? > > > > > We > > > > > > are working on the 2.5.1 release, which would be the recommended > > > > release. > > > > > > > > > > > > Ismael > > > > > > > > > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia < > > > > > sankalpbhati...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > I would like to know if there are any plans to release a 2.3.2 > > and > > > > > 2.4.2 > > > > > > > versions for Apache Kafka in the near future. I see there are > > some > > > > > issues > > > > > > > marked as fixed in these two versions > > > > > > > (https://tinyurl.com/ycdpz5cb). > > > > > > > However, I could not find a branch/tag corresponding to these > > > > versions > > > > > in > > > > > > > the github repository (https://github.com/apache/kafka). > > > > > > > > > > > > > > Also, It would be great if someone can help me understand or > > share > > > > any > > > > > > > documentation around the release processes (specifically on > when > > we > > > > > > decide > > > > > > > to release a new bug fix version like the ones mentioned > above.) > > > > > > > > > > > > > > Thanks, > > > > > > > Sankalp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Gwen Shapira > Engineering Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2
I agree that is an option, but is there any reason to not have a 2.3.2 and 2.4.2 released? If so, it would be nice to know about these reasons. Appreciate your time on this. On Wed, Jun 24, 2020 at 6:35 PM Ismael Juma wrote: > I think there's some confusion here. You can upgrade to AK 2.5.0 and > completely ignore ZK and TLS. It's completely optional. > > Ismael > > On Wed, Jun 24, 2020 at 10:20 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating > > from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it > would > > be nice to have a bug-free version of 2.3 and 2.4. > > > > Is there a technical reason why we hesitate to get these versions out? Or > > is it that no one has got around to it? > > > > On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia < > sankalpbhati...@gmail.com> > > wrote: > > > > > Thanks Ismael for the response. > > > > > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which had > > > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it > > > introduces some major changes like support for ZK encryption and a few > > > others, I was wondering if we should choose a smaller upgrade in such > > cases > > > as we don't really require the new features in 2.5 and above right now. > > > > > > - > > > Sankalp > > > > > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma wrote: > > > > > > > Hi Sankalp, > > > > > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 > instead? > > > We > > > > are working on the 2.5.1 release, which would be the recommended > > release. > > > > > > > > Ismael > > > > > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia < > > > sankalpbhati...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I would like to know if there are any plans to release a 2.3.2 and > > > 2.4.2 > > > > > versions for Apache Kafka in the near future. I see there are some > > > issues > > > > > marked as fixed in these two versions > > > > > (https://tinyurl.com/ycdpz5cb). > > > > > However, I could not find a branch/tag corresponding to these > > versions > > > in > > > > > the github repository (https://github.com/apache/kafka). > > > > > > > > > > Also, It would be great if someone can help me understand or share > > any > > > > > documentation around the release processes (specifically on when we > > > > decide > > > > > to release a new bug fix version like the ones mentioned above.) > > > > > > > > > > Thanks, > > > > > Sankalp > > > > > > > > > > > > > > >
Re: Question around release plans for Apache Kafka 2.3.2 and 2.4.2
Any updates on Kafka 2.3.2 and 2.4.2? Given the complexity of migrating from non-encrypted TLS to encrypted TLS connection for ZooKeeper, it would be nice to have a bug-free version of 2.3 and 2.4. Is there a technical reason why we hesitate to get these versions out? Or is it that no one has got around to it? On Wed, Jun 24, 2020 at 3:19 PM Sankalp Bhatia wrote: > Thanks Ismael for the response. > > For our clusters running 2.3.1 and 2.4.1, we saw some issues which had > 2.3.2 and 2.4.2 as the fix versions. I looked at 2.5.0 but since it > introduces some major changes like support for ZK encryption and a few > others, I was wondering if we should choose a smaller upgrade in such cases > as we don't really require the new features in 2.5 and above right now. > > - > Sankalp > > On Wed, 24 Jun 2020 at 14:23, Ismael Juma wrote: > > > Hi Sankalp, > > > > Is there a reason why you cannot upgrade to Apache Kafka 2.5.0 instead? > We > > are working on the 2.5.1 release, which would be the recommended release. > > > > Ismael > > > > On Wed, Jun 24, 2020 at 6:18 AM Sankalp Bhatia < > sankalpbhati...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > I would like to know if there are any plans to release a 2.3.2 and > 2.4.2 > > > versions for Apache Kafka in the near future. I see there are some > issues > > > marked as fixed in these two versions > > > (https://tinyurl.com/ycdpz5cb). > > > However, I could not find a branch/tag corresponding to these versions > in > > > the github repository (https://github.com/apache/kafka). > > > > > > Also, It would be great if someone can help me understand or share any > > > documentation around the release processes (specifically on when we > > decide > > > to release a new bug fix version like the ones mentioned above.) > > > > > > Thanks, > > > Sankalp > > > > > >
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Alex, 904. For multi-tenancy, yes, the limits in this KIP will apply on a first-come-first-serve basis. If and when multi-tenancy based limiting rolls out, it would probably introduce a different way to specific per-client limits on partitions. I suppose those will apply in addition to and not instead of the broker-level limits. 905. Yes, it is possible to have a scenario where the brokers in only one rack have partition capacity left (as imposed by KIP-578), and therefore, that all replicas for a new topic partition get assigned to that rack. But in practice, this would only be a problem if the limit was set to too low a value. In practice, the partition limits should be set to such a high value that when nearing it, there would be a risk of an availability loss. Therefore, I would treat replica assignment as we near the partition limits an edge case that should rarely happen in practice. Further, if a replica reassignment is not proving possible because a plan is considered invalid in light of the limits, and the administrator believes that the limit is too low, they can increase it (just like any other dynamic config) and retry the reassignment. 906. The case of internal topics and auto-topic creation will be addressed by KIP-590, which will modify the broker to send a request to the controller for topic creation rather than directly write to ZooKeeper. The fact that we don't handle internal topics is not a feature, it is a "bug". Therefore, there is no plan to support allowlist/denylists for topics that can simply bypass the partition limits. 907. If by update, you mean Kafka version upgrade, I have explained this in the migration plan. Basically, if the brokers have more partitions at a point in time when the partition limit is applied, then the broker will continue to work as expected. i.e. it will not check that it has more partitions than expected, and crash. Instead, it will simply start denying new requests to create partitions. In other words, having as many partitions as the limit is the same as having more partitions than the limit. On Fri, Jun 12, 2020 at 10:57 AM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Gokul, > > Thank you for the answers and the data provided to illustrate the use case. > A couple of additional questions. > > 904. If multi-tenancy is addressed in a future KIP, how smooth would > be the upgrade path? For example, the introduced configuration > parameters still apply, right? We would still maintain a first-come > first-served pattern when topics are created? > > 905. The current built-in assignment tool prioritises balance between > racks over brokers. In the version you propose, the limit on partition > count would take precedence over attempts to balance between racks. > Could it lead to a situation where it results in all partitions of a > topic being assigned in a single data center, if brokers in other > racks are "full"? Since it can potentially weaken the availability > guarantees for that topic (and maybe durability and/or consumer > performance with additional cross-rack traffic), how would we want to > handle the case? It may be worth warning users that the resulting > guarantees differ from what is offered by an "unlimited" assignment > plan in such cases? Also, let's keep in mind that some plans generated > by existing rebalancing tools could become invalid (w.r.t to the > configured limits). > > 906. The limits do not apply to internal topics. What about > framework-level topics from other tools and extensions? (connect, > streams, confluent metrics, tiered storage, etc.) Is blacklisting > possible? > > 907. What happens if one of the dynamic limit is violated at update > time? (sorry if it's already explained in the KIP, may have missed it) > > Thanks, > Alexandre > > Le dim. 3 mai 2020 à 20:20, Gokul Ramanan Subramanian > a écrit : > > > > Thanks Stanislav. Apologies about the long absence from this thread. > > > > I would prefer having per-user max partition limits in a separate KIP. I > > don't see this as an MVP for this KIP. I will add this as an alternative > > approach into the KIP. > > > > I was in a double mind about whether or not to impose the partition limit > > for internal topics as well. I can be convinced both ways. On the one > hand, > > internal topics should be purely internal i.e. users of a cluster should > > not have to care about them. In this sense, the partition limit should > not > > apply to internal topics. On the other hand, Kafka allows configuring > > internal topics by specifying their replication factor etc. Therefore, > they > > don't feel all that internal to me. In any case, I'll modify the KIP to > > exclude internal topics. > > > > I'l
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Boyang, Tom and I discussed earlier in this thread whether we could use the topic creation policy for limiting the number of partitions. Basically, we realized that while we can use it for limiting the number of partitions, we cannot use it for assigning partitions in a way that respects the limit. In light of KIP-590, I am thinking to modify KIP-578 to just say that we won't apply the partition limit in case of auto-topic creation wherein the Metadata API directly writes to ZK. Once KIP-590 surfaces, users will get partition limiting during auto-topic creation naturally without any changes required, because Metadata API will call CreateTopics under the covers. What do you think? I have updated the KIP to indicate that we will bump up the versions for the affected APIs. For older clients, I was assuming that if we throw the new exception from the new broker (with the KIP-578 changes), and it gets sent to an older client which cannot interpret the exception, it will be converted to some generic exception. Is this not the case? I am not sure I understand a way to send another type of exception just for older clients. Could you kindly explain. Also, all this might be simplified if we just reuse the existing PolicyViolation exception as Ismael suggested. Thanks. On Wed, Jun 24, 2020 at 3:26 AM Boyang Chen wrote: > Hi Gokul, > > Thanks for the excellent KIP. I was recently driving the rollout of KIP-590 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller > > > and > proposed to fix the hole of the bypassing of topic creation policy when > applying Metadata auto topic creation. As far as I understand, this KIP > will take care of this problem specifically towards partition limit, so I > was wondering whether it makes sense to integrate this logic with topic > creation policy, and just enforce that entirely on the Metadata RPC auto > topic creation case? > > Also, for the API exception part, could we state clearly that we are going > to bump the version of mentioned RPCs so that new clients could handle the > new error code RESOURCE_LIMITE_REACHED? Besides, I believe we are also > going to enforce this rule towards older clients, could you describe the > corresponding error code to be returned in the API exception part as well? > > Let me know if my proposal makes sense, thank you! > > Boyang > > On Fri, Jun 12, 2020 at 2:57 AM Alexandre Dupriez < > alexandre.dupr...@gmail.com> wrote: > > > Hi Gokul, > > > > Thank you for the answers and the data provided to illustrate the use > case. > > A couple of additional questions. > > > > 904. If multi-tenancy is addressed in a future KIP, how smooth would > > be the upgrade path? For example, the introduced configuration > > parameters still apply, right? We would still maintain a first-come > > first-served pattern when topics are created? > > > > 905. The current built-in assignment tool prioritises balance between > > racks over brokers. In the version you propose, the limit on partition > > count would take precedence over attempts to balance between racks. > > Could it lead to a situation where it results in all partitions of a > > topic being assigned in a single data center, if brokers in other > > racks are "full"? Since it can potentially weaken the availability > > guarantees for that topic (and maybe durability and/or consumer > > performance with additional cross-rack traffic), how would we want to > > handle the case? It may be worth warning users that the resulting > > guarantees differ from what is offered by an "unlimited" assignment > > plan in such cases? Also, let's keep in mind that some plans generated > > by existing rebalancing tools could become invalid (w.r.t to the > > configured limits). > > > > 906. The limits do not apply to internal topics. What about > > framework-level topics from other tools and extensions? (connect, > > streams, confluent metrics, tiered storage, etc.) Is blacklisting > > possible? > > > > 907. What happens if one of the dynamic limit is violated at update > > time? (sorry if it's already explained in the KIP, may have missed it) > > > > Thanks, > > Alexandre > > > > Le dim. 3 mai 2020 à 20:20, Gokul Ramanan Subramanian > > a écrit : > > > > > > Thanks Stanislav. Apologies about the long absence from this thread. > > > > > > I would prefer having per-user max partition limits in a separate KIP. > I > > > don't see this as an MVP for this KIP. I will add this as an > alternative > > > approach into the KIP. > > > > > > I was in
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Ismael, I am open to using any error code and am not attached to one TBH. Colin had suggested creating a new resource code called RESOURCE_LIMIT_EXCEEDED. I am happy to reuse the error code corresponding to PolicyViolation. Is it safe to rename errors and corresponding exception names? If so, I'd prefer reusing the existing code as well. For performance testing results that I added to this KIP, I used Kafka 2.3.1, which was very close to trunk at the time I tested. We have seen similar issues with Kafka 2.4.1. Please note that for the tests done in the KIP, especially the Produce performance tests, we probably could have gotten higher performance, but the focus was on comparing performance across a different number of partitions, for a given configuration, rather than trying to find out the best performance possible for the "right" number of partitions. Thanks. On Wed, Jun 24, 2020 at 4:07 AM Ismael Juma wrote: > Thanks for the KIP. A couple of questions: > > 1. Have we considered reusing the existing PolicyViolation error code and > renaming it? This would make it simpler to handle on the client. > > 2. What version was used for the perf section? I think master should do > better than what's described there. > > Ismael > > On Wed, Apr 1, 2020, 8:28 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> > wrote: > > > Hi. > > > > I have opened KIP-578, intended to provide a mechanism to limit the > number > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP > which > > you can find at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > > I want to specially thank Stanislav Kozlovski who helped in formulating > > some aspects of the KIP. > > > > Many thanks, > > > > Gokul. > > >
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi. Can we resume the voting process for KIP-578? Thanks. On Mon, Jun 1, 2020 at 11:09 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Thanks Colin. Have updated the KIP per your recommendations. Let me know > what you think. > > Thanks Harsha for the vote. > > On Wed, May 27, 2020 at 8:17 PM Colin McCabe wrote: > >> Hi Gokul Ramanan Subramanian, >> >> Thanks for the KIP. >> >> Can you please modify the KIP to remove the reference to the deprecated >> --zookeeper flag? This is not how kafka-configs.sh is supposed to be used >> in new versions of Kafka. You get a warning message if you do use this >> deprecated flag. As described in KIP-604, we are removing the --zookeeper >> flag in the Kafka 3.0 release. It also causes problems when people use the >> deprecated access mode-- for example, as you note in this KIP, it bypasses >> resource limits such as the ones described here. >> >> Instead of WILL_EXCEED_PARTITION_LIMITS, how about >> RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed >> message about which resource limit was hit (per broker limit, per cluster >> limit, whatever.) It would also be good to spell out that >> CreateTopicsPolicy plugins can also throw this exception, for consistency. >> >> I realize that 2 billion partitions seems like a very big number. >> However, filesystems have had to transition to 64 bit inode numbers as time >> has gone on. There doesn't seem to be any performance reason why this >> should be a 31 bit number, so let's just make these configurations longs, >> not ints. >> >> best, >> Colin >> >> >> On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: >> > Thanks for the KIP Gokul. This will be really useful for our use cases >> as >> > well. >> > +1 (binding). >> > >> > -Harsha >> > >> > >> > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < >> > gokul24...@gmail.com> wrote: >> > >> > > Hi. >> > > >> > > Any votes for this? >> > > >> > > Thanks. >> > > >> > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < >> gokul2411s@ >> > > gmail.com> wrote: >> > > >> > > Hello, >> > > >> > > I'd like to initialize voting on KIP-578: >> > > https://cwiki.apache.org/confluence/display/KAFKA/ >> > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions >> > > . >> > > >> > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez >> and Tom >> > > Bentley on the discussion thread. I have addressed their comments. I >> want >> > > to thank them for their time. >> > > >> > > If there are any more concerns about the KIP, I am happy to discuss >> them >> > > further. >> > > >> > > Thanks. >> > > >> > > >> > >> >
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Thanks Colin. Have updated the KIP per your recommendations. Let me know what you think. Thanks Harsha for the vote. On Wed, May 27, 2020 at 8:17 PM Colin McCabe wrote: > Hi Gokul Ramanan Subramanian, > > Thanks for the KIP. > > Can you please modify the KIP to remove the reference to the deprecated > --zookeeper flag? This is not how kafka-configs.sh is supposed to be used > in new versions of Kafka. You get a warning message if you do use this > deprecated flag. As described in KIP-604, we are removing the --zookeeper > flag in the Kafka 3.0 release. It also causes problems when people use the > deprecated access mode-- for example, as you note in this KIP, it bypasses > resource limits such as the ones described here. > > Instead of WILL_EXCEED_PARTITION_LIMITS, how about > RESOURCE_LIMIT_REACHED? Then the error string can contain the detailed > message about which resource limit was hit (per broker limit, per cluster > limit, whatever.) It would also be good to spell out that > CreateTopicsPolicy plugins can also throw this exception, for consistency. > > I realize that 2 billion partitions seems like a very big number. > However, filesystems have had to transition to 64 bit inode numbers as time > has gone on. There doesn't seem to be any performance reason why this > should be a 31 bit number, so let's just make these configurations longs, > not ints. > > best, > Colin > > > On Wed, May 27, 2020, at 09:48, Harsha Chintalapani wrote: > > Thanks for the KIP Gokul. This will be really useful for our use cases as > > well. > > +1 (binding). > > > > -Harsha > > > > > > On Tue, May 26, 2020 at 12:33 AM, Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Hi. > > > > > > Any votes for this? > > > > > > Thanks. > > > > > > On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < > gokul2411s@ > > > gmail.com> wrote: > > > > > > Hello, > > > > > > I'd like to initialize voting on KIP-578: > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > . > > > > > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez and > Tom > > > Bentley on the discussion thread. I have addressed their comments. I > want > > > to thank them for their time. > > > > > > If there are any more concerns about the KIP, I am happy to discuss > them > > > further. > > > > > > Thanks. > > > > > > > > >
Re: [VOTE] KIP-578: Add configuration to limit number of partitions
Hi. Any votes for this? Thanks. On Tue, May 12, 2020 at 11:36 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hello, > > I'd like to initialize voting on KIP-578: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > . > > Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez and Tom > Bentley on the discussion thread. I have addressed their comments. I want > to thank them for their time. > > If there are any more concerns about the KIP, I am happy to discuss them > further. > > Thanks. >
[VOTE] KIP-578: Add configuration to limit number of partitions
Hello, I'd like to initialize voting on KIP-578: https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions . Got some good feedback from Stanislav Kozlovski, Alexandre Dupriez and Tom Bentley on the discussion thread. I have addressed their comments. I want to thank them for their time. If there are any more concerns about the KIP, I am happy to discuss them further. Thanks.
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Thanks Stanislav. Apologies about the long absence from this thread. I would prefer having per-user max partition limits in a separate KIP. I don't see this as an MVP for this KIP. I will add this as an alternative approach into the KIP. I was in a double mind about whether or not to impose the partition limit for internal topics as well. I can be convinced both ways. On the one hand, internal topics should be purely internal i.e. users of a cluster should not have to care about them. In this sense, the partition limit should not apply to internal topics. On the other hand, Kafka allows configuring internal topics by specifying their replication factor etc. Therefore, they don't feel all that internal to me. In any case, I'll modify the KIP to exclude internal topics. I'll also add to the KIP the alternative approach Tom suggested around using topic policies to limit partitions, and explain why it does not help to solve the problem that the KIP is trying to address (as I have done in a previous correspondence on this thread). Cheers. On Fri, Apr 24, 2020 at 4:24 PM Stanislav Kozlovski wrote: > Thanks for the KIP, Gokul! > > I like the overall premise - I think it's more user-friendly to have > configs for this than to have users implement their own config policy -> so > unless it's very complex to implement, it seems worth it. > I agree that having the topic policy on the CreatePartitions path makes > sense as well. > > Multi-tenancy was a good point. It would be interesting to see how easy it > is to extend the max partition limit to a per-user basis. Perhaps this can > be done in a follow-up KIP, as a natural extension of the feature. > > I'm wondering whether there's a need to enforce this on internal topics, > though. Given they're internal and critical to the function of Kafka, I > believe we'd rather always ensure they're created, regardless if over some > user-set limit. It brings up the question of forward compatibility - what > happens if a user's cluster is at the maximum partition capacity, yet a new > release of Kafka introduces a new topic (e.g KIP-500)? > > Best, > Stanislav > > On Fri, Apr 24, 2020 at 2:39 PM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi Tom. > > > > With KIP-578, we are not trying to model the load on each partition, and > > come up with an exact limit on what the cluster or broker can handle in > > terms of number of partitions. We understand that not all partitions are > > equal, and the actual load per partition varies based on the message > size, > > throughput, whether the broker is a leader for that partition or not etc. > > > > What we are trying to achieve with KIP-578 is to disallow a pathological > > number of partitions that will surely put the cluster in bad shape. For > > example, in KIP-578's appendix, we have described a case where we could > not > > delete a topic with 30k partitions, because the brokers could not > > handle all the work that needed to be done. We have also described how > > a producer performance test with 10k partitions observed basically 0 > > throughput. In these cases, having a limit on number of partitions > > would allow the cluster to produce a graceful error message at topic > > creation time, and prevent the cluster from entering a pathological > state. > > These are not just hypotheticals. We definitely see many of these > > pathological cases happen in production, and we would like to avoid them. > > > > The actual limit on number of partitions is something we do not want to > > suggest in the KIP. The limit will depend on various tests that owners of > > their clusters will have to perform, including perf tests, identifying > > topic creation / deletion times, etc. For example, the tests we did for > the > > KIP-578 appendix were enough to convince us that we should not have > > anywhere close to 10k partitions on the setup we describe there. > > > > What we want to do with KIP-578 is provide the flexibility to set a limit > > on number of partitions based on tests cluster owners choose to perform. > > Cluster owners can do the tests however often they wish and dynamically > > adjust the limit on number of partitions. For example, we found in our > > production environment that we don't want to have more than 1k partitions > > on an m5.large EC2 broker instances, or more than 300 partitions on a > > t3.medium EC2 broker, for typical produce / consume use cases. > > > > Cluster owners are free to not configure the limit on number of > partitions > > if they don't want to spend the time coming up with a limit. The limit > > defaults to INT32_MAX, which is basically infinity i
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
imisation algorithm for figuring out > assignments of partitions to brokers, and it could be that the best (or at > least a good enough) solution requires assigning the least loaded 41 > partitions to one broker. > > The point I'm trying to make here is whatever limit is chosen it's probably > been chosen fairly arbitrarily. Even if it's been chosen based on some > empirical evidence of how a particular cluster behaves it's likely that > that evidence will become obsolete as the cluster evolves to serve the > needs of the business running it (e.g. some hot topic gets repartitioned, > messages get compressed with some new algorithm, some new topics need to be > created). For this reason I think the problem you're trying to solve via > policy (whether that was implemented in a pluggable way or not) is really > better solved by automating the cluster balancing and having that cluster > balancer be able to reason about when the cluster has too few brokers for > the number of partitions, rather than placing some limit on the sizing and > shape of the cluster up front and then hobbling the cluster balancer to > work within that. > > I think it might be useful to describe in the KIP how users would be > expected to arrive at values for these configs (both on day 1 and in an > evolving production cluster), when this solution might be better than using > a cluster balancer and/or why cluster balancers can't be trusted to avoid > overloading brokers. > > Kind regards, > > Tom > > > On Mon, Apr 20, 2020 at 7:22 PM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > This is good reference Tom. I did not consider this approach at all. I am > > happy to learn about it now. > > > > However, I think that partition limits are not "yet another" policy > > configuration. Instead, they are fundamental to partition assignment. > i.e. > > the partition assignment algorithm needs to be aware of the partition > > limits. To illustrate this, imagine that you have 3 brokers (1, 2 and 3), > > with 10, 20 and 30 partitions each respectively, and a limit of 40 > > partitions on each broker enforced via a configurable policy class (the > one > > you recommended). While the policy class may accept a topic creation > > request for 11 partitions with a replication factor of 2 each (because it > > is satisfiable), the non-pluggable partition assignment algorithm (in > > AdminUtils.assignReplicasToBrokers and a few other places) has to know > not > > to assign the 11th partition to broker 3 because it would run out of > > partition capacity otherwise. > > > > To achieve the ideal end that you are imagining (and I can totally > > understand where you are coming from vis-a-vis the extensibility of your > > solution wrt the one in the KIP), that would require extracting the > > partition assignment logic itself into a pluggable class, and for which > we > > could provide a custom implementation. I am afraid that would add > > complexity that I am not sure we want to undertake. > > > > Do you see sense in what I am saying? > > > > Thanks. > > > > On Mon, Apr 20, 2020 at 3:59 PM Tom Bentley wrote: > > > > > Hi Gokul, > > > > > > Leaving aside the question of how Kafka scales, I think the proposed > > > solution, limiting the number of partitions in a cluster or per-broker, > > is > > > a policy which ought to be addressable via the pluggable policies (e.g. > > > create.topic.policy.class.name). Unfortunately although there's a > policy > > > for topic creation, it's currently not possible to enforce a policy on > > > partition increase. It would be more flexible to be able enforce this > > kind > > > of thing via a pluggable policy, and it would also avoid the situation > > > where different people each want to have a config which addresses some > > > specific use case or problem that they're experiencing. > > > > > > Quite a while ago I proposed KIP-201 to solve this issue with policies > > > being easily circumvented, but it didn't really make any progress. I've > > > looked at it again in some detail more recently and I think something > > might > > > be possible following the work to make all ZK writes happen on the > > > controller. > > > > > > Of course, this is just my take on it. > > > > > > Kind regards, > > > > > > Tom > > > > > > On Thu, Apr 16, 2020 at 11:47 AM Gokul Ramanan Subramanian < > > > gokul24...@gmail.com> wrote: > > > > > >
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
This is good reference Tom. I did not consider this approach at all. I am happy to learn about it now. However, I think that partition limits are not "yet another" policy configuration. Instead, they are fundamental to partition assignment. i.e. the partition assignment algorithm needs to be aware of the partition limits. To illustrate this, imagine that you have 3 brokers (1, 2 and 3), with 10, 20 and 30 partitions each respectively, and a limit of 40 partitions on each broker enforced via a configurable policy class (the one you recommended). While the policy class may accept a topic creation request for 11 partitions with a replication factor of 2 each (because it is satisfiable), the non-pluggable partition assignment algorithm (in AdminUtils.assignReplicasToBrokers and a few other places) has to know not to assign the 11th partition to broker 3 because it would run out of partition capacity otherwise. To achieve the ideal end that you are imagining (and I can totally understand where you are coming from vis-a-vis the extensibility of your solution wrt the one in the KIP), that would require extracting the partition assignment logic itself into a pluggable class, and for which we could provide a custom implementation. I am afraid that would add complexity that I am not sure we want to undertake. Do you see sense in what I am saying? Thanks. On Mon, Apr 20, 2020 at 3:59 PM Tom Bentley wrote: > Hi Gokul, > > Leaving aside the question of how Kafka scales, I think the proposed > solution, limiting the number of partitions in a cluster or per-broker, is > a policy which ought to be addressable via the pluggable policies (e.g. > create.topic.policy.class.name). Unfortunately although there's a policy > for topic creation, it's currently not possible to enforce a policy on > partition increase. It would be more flexible to be able enforce this kind > of thing via a pluggable policy, and it would also avoid the situation > where different people each want to have a config which addresses some > specific use case or problem that they're experiencing. > > Quite a while ago I proposed KIP-201 to solve this issue with policies > being easily circumvented, but it didn't really make any progress. I've > looked at it again in some detail more recently and I think something might > be possible following the work to make all ZK writes happen on the > controller. > > Of course, this is just my take on it. > > Kind regards, > > Tom > > On Thu, Apr 16, 2020 at 11:47 AM Gokul Ramanan Subramanian < > gokul24...@gmail.com> wrote: > > > Hi. > > > > For the sake of expediting the discussion, I have created a prototype PR: > > https://github.com/apache/kafka/pull/8499. Eventually, (if and) when the > > KIP is accepted, I'll modify this to add the full implementation and > tests > > etc. in there. > > > > Would appreciate if a Kafka committer could share their thoughts, so > that I > > can more confidently start the voting thread. > > > > Thanks. > > > > On Thu, Apr 16, 2020 at 11:30 AM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Thanks for your comments Alex. > > > > > > The KIP proposes using two configurations max.partitions and > > > max.broker.partitions. It does not enforce their use. The default > values > > > are pretty large (INT MAX), therefore should be non-intrusive. > > > > > > In multi-tenant environments and in partition assignment and > rebalancing, > > > the admin could (a) use the default values which would yield similar > > > behavior to now, (b) set very high values that they know is sufficient, > > (c) > > > dynamically re-adjust the values should the business requirements > change. > > > Note that the two configurations are cluster-wide, so they can be > updated > > > without restarting the brokers. > > > > > > The quota system in Kafka seems to be geared towards limiting traffic > for > > > specific clients or users, or in the case of replication, to leaders > and > > > followers. The quota configuration itself is very similar to the one > > > introduced in this KIP i.e. just a few configuration options to specify > > the > > > quota. The main difference is that the quota system is far more > > > heavy-weight because it needs to be applied to traffic that is flowing > > > in/out constantly. Whereas in this KIP, we want to limit number of > > > partition replicas, which gets modified rarely by comparison in a > typical > > > cluster. > > > > > > Hope this addresses your comments. > > > > > > On Thu, Apr 9,
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Hi. For the sake of expediting the discussion, I have created a prototype PR: https://github.com/apache/kafka/pull/8499. Eventually, (if and) when the KIP is accepted, I'll modify this to add the full implementation and tests etc. in there. Would appreciate if a Kafka committer could share their thoughts, so that I can more confidently start the voting thread. Thanks. On Thu, Apr 16, 2020 at 11:30 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Thanks for your comments Alex. > > The KIP proposes using two configurations max.partitions and > max.broker.partitions. It does not enforce their use. The default values > are pretty large (INT MAX), therefore should be non-intrusive. > > In multi-tenant environments and in partition assignment and rebalancing, > the admin could (a) use the default values which would yield similar > behavior to now, (b) set very high values that they know is sufficient, (c) > dynamically re-adjust the values should the business requirements change. > Note that the two configurations are cluster-wide, so they can be updated > without restarting the brokers. > > The quota system in Kafka seems to be geared towards limiting traffic for > specific clients or users, or in the case of replication, to leaders and > followers. The quota configuration itself is very similar to the one > introduced in this KIP i.e. just a few configuration options to specify the > quota. The main difference is that the quota system is far more > heavy-weight because it needs to be applied to traffic that is flowing > in/out constantly. Whereas in this KIP, we want to limit number of > partition replicas, which gets modified rarely by comparison in a typical > cluster. > > Hope this addresses your comments. > > On Thu, Apr 9, 2020 at 12:53 PM Alexandre Dupriez < > alexandre.dupr...@gmail.com> wrote: > >> Hi Gokul, >> >> Thanks for the KIP. >> >> From what I understand, the objective of the new configuration is to >> protect a cluster from an overload driven by an excessive number of >> partitions independently from the load handled on the partitions >> themselves. As such, the approach uncouples the data-path load from >> the number of unit of distributions of throughput and intends to avoid >> the degradation of performance exhibited in the test results provided >> with the KIP by setting an upper-bound on that number. >> >> Couple of comments: >> >> 900. Multi-tenancy - one concern I would have with a cluster and >> broker-level configuration is that it is possible for a user to >> consume a large proportions of the allocatable partitions within the >> configured limit, leaving other users with not enough partitions to >> satisfy their requirements. >> >> 901. Quotas - an approach in Apache Kafka to set-up an upper-bound on >> resource consumptions is via client/user quotas. Could this framework >> be leveraged to add this limit? >> >> 902. Partition assignment - one potential problem with the new >> repartitioning scheme is that if a subset of brokers have reached >> their number of assignable partitions, yet their data path is >> under-loaded, new topics and/or partitions will be assigned >> exclusively to other brokers, which could increase the likelihood of >> data-path load imbalance. Fundamentally, the isolation of the >> constraint on the number of partitions from the data-path throughput >> can have conflicting requirements. >> >> 903. Rebalancing - as a corollary to 902, external tools used to >> balance ingress throughput may adopt an incremental approach in >> partition re-assignment to redistribute load, and could hit the limit >> on the number of partitions on a broker when a (too) conservative >> limit is used, thereby over-constraining the objective function and >> reducing the migration path. >> >> Thanks, >> Alexandre >> >> Le jeu. 9 avr. 2020 à 00:19, Gokul Ramanan Subramanian >> a écrit : >> > >> > Hi. Requesting you to take a look at this KIP and provide feedback. >> > >> > Thanks. Regards. >> > >> > On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < >> > gokul24...@gmail.com> wrote: >> > >> > > Hi. >> > > >> > > I have opened KIP-578, intended to provide a mechanism to limit the >> number >> > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP >> which >> > > you can find at >> > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions >> > > >> > > I want to specially thank Stanislav Kozlovski who helped in >> formulating >> > > some aspects of the KIP. >> > > >> > > Many thanks, >> > > >> > > Gokul. >> > > >> >
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Thanks for your comments Alex. The KIP proposes using two configurations max.partitions and max.broker.partitions. It does not enforce their use. The default values are pretty large (INT MAX), therefore should be non-intrusive. In multi-tenant environments and in partition assignment and rebalancing, the admin could (a) use the default values which would yield similar behavior to now, (b) set very high values that they know is sufficient, (c) dynamically re-adjust the values should the business requirements change. Note that the two configurations are cluster-wide, so they can be updated without restarting the brokers. The quota system in Kafka seems to be geared towards limiting traffic for specific clients or users, or in the case of replication, to leaders and followers. The quota configuration itself is very similar to the one introduced in this KIP i.e. just a few configuration options to specify the quota. The main difference is that the quota system is far more heavy-weight because it needs to be applied to traffic that is flowing in/out constantly. Whereas in this KIP, we want to limit number of partition replicas, which gets modified rarely by comparison in a typical cluster. Hope this addresses your comments. On Thu, Apr 9, 2020 at 12:53 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Gokul, > > Thanks for the KIP. > > From what I understand, the objective of the new configuration is to > protect a cluster from an overload driven by an excessive number of > partitions independently from the load handled on the partitions > themselves. As such, the approach uncouples the data-path load from > the number of unit of distributions of throughput and intends to avoid > the degradation of performance exhibited in the test results provided > with the KIP by setting an upper-bound on that number. > > Couple of comments: > > 900. Multi-tenancy - one concern I would have with a cluster and > broker-level configuration is that it is possible for a user to > consume a large proportions of the allocatable partitions within the > configured limit, leaving other users with not enough partitions to > satisfy their requirements. > > 901. Quotas - an approach in Apache Kafka to set-up an upper-bound on > resource consumptions is via client/user quotas. Could this framework > be leveraged to add this limit? > > 902. Partition assignment - one potential problem with the new > repartitioning scheme is that if a subset of brokers have reached > their number of assignable partitions, yet their data path is > under-loaded, new topics and/or partitions will be assigned > exclusively to other brokers, which could increase the likelihood of > data-path load imbalance. Fundamentally, the isolation of the > constraint on the number of partitions from the data-path throughput > can have conflicting requirements. > > 903. Rebalancing - as a corollary to 902, external tools used to > balance ingress throughput may adopt an incremental approach in > partition re-assignment to redistribute load, and could hit the limit > on the number of partitions on a broker when a (too) conservative > limit is used, thereby over-constraining the objective function and > reducing the migration path. > > Thanks, > Alexandre > > Le jeu. 9 avr. 2020 à 00:19, Gokul Ramanan Subramanian > a écrit : > > > > Hi. Requesting you to take a look at this KIP and provide feedback. > > > > Thanks. Regards. > > > > On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < > > gokul24...@gmail.com> wrote: > > > > > Hi. > > > > > > I have opened KIP-578, intended to provide a mechanism to limit the > number > > > of partitions in a Kafka cluster. Kindly provide feedback on the KIP > which > > > you can find at > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > > > > > I want to specially thank Stanislav Kozlovski who helped in formulating > > > some aspects of the KIP. > > > > > > Many thanks, > > > > > > Gokul. > > > >
Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions
Hi. Requesting you to take a look at this KIP and provide feedback. Thanks. Regards. On Wed, Apr 1, 2020 at 4:28 PM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi. > > I have opened KIP-578, intended to provide a mechanism to limit the number > of partitions in a Kafka cluster. Kindly provide feedback on the KIP which > you can find at > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions > > I want to specially thank Stanislav Kozlovski who helped in formulating > some aspects of the KIP. > > Many thanks, > > Gokul. >
[DISCUSS] KIP-578: Add configuration to limit number of partitions
Hi. I have opened KIP-578, intended to provide a mechanism to limit the number of partitions in a Kafka cluster. Kindly provide feedback on the KIP which you can find at https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions I want to specially thank Stanislav Kozlovski who helped in formulating some aspects of the KIP. Many thanks, Gokul.
Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor
+1 (non binding). Thanks. On Tue, Mar 31, 2020 at 2:43 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > +1 (non binding). > > Thanks. > > Le mar. 31 mars 2020 à 14:24, M. Manna a écrit : > > > > +1 (binding). > > > > Thanks for the KIP. > > > > On Tue, 31 Mar 2020 at 14:17, Paolo Moriello > > wrote: > > > > > Hello, > > > > > > Thanks to everybody who has given feedback. I've incorporated the > > > suggestions and think that this is now ready for a vote. > > > > > > KIP 579: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor > > > > > > PR: > > > https://github.com/apache/kafka/pull/8225 > > > > > > Thanks, > > > Paolo > > > >
Has restarted broker caught up?
Hi. What is the idiomatic / correct way to know that a restarted broker in an existing Kafka cluster has caught up to all the leaders for the partitions it hosts? Is there a specific JMX metric published either by the leader / follower that we can monitor? Ideally, the metric would have the following characteristics: *Metric:* IsInSync=Yes/No *Dimensions:* LeaderId = L, FollowerId = F, Topic = T, Partition = P If not, do you think this would be useful?
Limit number of topic partitions
Hi. I'd like your thoughts on https://issues.apache.org/jira/browse/KAFKA-9590?jql=text%20~%20%22limit%20topic%20partitions%22 (tldr: its about limiting number of topic partitions via a read-only configuration). I am going to start writing a KIP for this soon, and will share on this thread as I make progress. But I thought it would be a good idea to get some initial thoughts / additional requirements (if any) from the community before I dive deep into the design. Thanks for your time.
[jira] [Created] (KAFKA-9590) Add read-only configuration to limit number of topic partitions
Gokul Ramanan Subramanian created KAFKA-9590: Summary: Add read-only configuration to limit number of topic partitions Key: KAFKA-9590 URL: https://issues.apache.org/jira/browse/KAFKA-9590 Project: Kafka Issue Type: Improvement Components: admin Reporter: Gokul Ramanan Subramanian Currently, a Kafka cluster does not limit the number of topic partitions despite the fact that on a given system, having more than a certain number of topic partitions (without even any messages written / read) renders the cluster unusable. The ask here is to have a read-only Kafka configuration named something like *max.topic.partitions* (an integer, which applies at the broker level and defaults to INT.MAX) that the administrator the cluster can set to ensure smooth operation. Things to keep in mind: # If a broker already has more than *max.topic.partitions* number of partitions at the time when the *max.topic.partitions* configuration is first applied on the broker, the broker should disallow any more topic partitions. This allows for graceful reassignments and in-place upgrade to a Kafka version that supports the *max.topic.partitions* configuration. # Different brokers can technically have different *max.topic.partitions* in case of a heterogenous cluster, where not all broker nodes operate with the same system limits (such as CPU, memory etc.) # When creating a topic partition, the controller has to identify brokers which will replicate the topic partition. Currently, the controller distributes the leadership of the topic partitions to ensure uniform load. This algorithm will need to account for the *max.topic.partitions* configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric
t.getEpochSecond(); > > > } > > > > > > ... > > > } > > > > > > Regards, > > > Sean > > > > > > > > > On Thu, Dec 12, 2019 at 6:18 AM Habib Nahas wrote: > > > > > >> Hi Sean, > > >> > > >> Thanks for the KIP. > > >> > > >> As I understand it users are free to set their own timestamp on > > >> ProducerRecord. What is the recommendation for the proposed metric in > a > > >> scenario where the user sets this timestamp in timezone A and > consumes the > > >> record in timezone B. Its not clear to me if a custom implementation > of > > >> LatencyTime will help here. > > >> > > >> Thanks, > > >> Habib > > >> > > >> On Wed, Dec 11, 2019, at 4:52 PM, Sean Glover wrote: > > >> > Hello again, > > >> > > > >> > There has been some interest in this KIP recently. I'm bumping the > > >> thread > > >> > to encourage feedback on the design. > > >> > > > >> > Regards, > > >> > Sean > > >> > > > >> > On Mon, Jul 15, 2019 at 9:01 AM Sean Glover < > sean.glo...@lightbend.com> > > >> > wrote: > > >> > > > >> > > To hopefully spark some discussion I've copied the motivation > section > > >> from > > >> > > the KIP: > > >> > > > > >> > > Consumer lag is a useful metric to monitor how many records are > > >> queued to > > >> > > be processed. We can look at individual lag per partition or we > may > > >> > > aggregate metrics. For example, we may want to monitor what the > > >> maximum lag > > >> > > of any particular partition in our consumer subscription so we can > > >> identify > > >> > > hot partitions, caused by an insufficient producing partitioning > > >> strategy. > > >> > > We may want to monitor a sum of lag across all partitions so we > have a > > >> > > sense as to our total backlog of messages to consume. Lag in > offsets > > >> is > > >> > > useful when you have a good understanding of your messages and > > >> processing > > >> > > characteristics, but it doesn’t tell us how far behind *in time* > we > > >> are. > > >> > > This is known as wait time in queueing theory, or more informally > it’s > > >> > > referred to as latency. > > >> > > > > >> > > The latency of a message can be defined as the difference between > when > > >> > > that message was first produced to when the message is received > by a > > >> > > consumer. The latency of records in a partition correlates with > lag, > > >> but a > > >> > > larger lag doesn’t necessarily mean a larger latency. For > example, a > > >> topic > > >> > > consumed by two separate application consumer groups A and B may > have > > >> > > similar lag, but different latency per partition. Application A > is a > > >> > > consumer which performs CPU intensive business logic on each > message > > >> it > > >> > > receives. It’s distributed across many consumer group members to > > >> handle the > > >> > > load quickly enough, but since its processing time is slower, it > takes > > >> > > longer to process each message per partition. Meanwhile, > Application > > >> B is > > >> > > a consumer which performs a simple ETL operation to land streaming > > >> data in > > >> > > another system, such as HDFS. It may have similar lag to > Application > > >> A, but > > >> > > because it has a faster processing time its latency per partition > is > > >> > > significantly less. > > >> > > > > >> > > If the Kafka Consumer reported a latency metric it would be > easier to > > >> > > build Service Level Agreements (SLAs) based on non-functional > > >> requirements > > >> > > of the streaming system. For example, the system must never have a > > >> latency > > >> > > of greater than 10 minutes. This SLA could be used in monitoring > > >> alerts or > > >> > > as input to automatic scaling solutions. > > >> > > > > >> > > On Thu, Jul 11, 2019 at 12:36 PM Sean Glover < > > >> sean.glo...@lightbend.com> > > >> > > wrote: > > >> > > > > >> > >> Hi kafka-dev, > > >> > >> > > >> > >> I've created KIP-489 as a proposal for adding latency metrics to > the > > >> > >> Kafka Consumer in a similar way as record-lag metrics are > > >> implemented. > > >> > >> > > >> > >> > > >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/489%3A+Kafka+Consumer+Record+Latency+Metric > > >> > >> > > >> > >> Regards, > > >> > >> Sean > > >> > >> > > >> > >> -- > > >> > >> Principal Engineer, Lightbend, Inc. > > >> > >> > > >> > >> <http://lightbend.com> > > >> > >> > > >> > >> @seg1o <https://twitter.com/seg1o>, in/seanaglover > > >> > >> <https://www.linkedin.com/in/seanaglover/> > > >> > >> > > >> > > > > >> > > > > >> > > -- > > >> > > Principal Engineer, Lightbend, Inc. > > >> > > > > >> > > <http://lightbend.com> > > >> > > > > >> > > @seg1o <https://twitter.com/seg1o>, in/seanaglover > > >> > > <https://www.linkedin.com/in/seanaglover/> > > >> > > > > >> > > > >> > > > > > > > > > > -- > > Sean Glover > > Principal Engineer, Alpakka, Lightbend, Inc. <https://lightbend.com> > > @seg1o <https://twitter.com/seg1o>, in/seanaglover > > <https://www.linkedin.com/in/seanaglover/> > > > -- Gokul Ramanan Subramanian Senior SDE, Amazon AWS
Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM
alue, and throw > > an InvalidReceiveException in a similar fashion as what happens on the > > broker, where a non-unlimited maxSize can be provided. Basically, mimic > the > > behaviour on the broker. The default value for the maximal request size > on > > the broker is 100 MB which you are suggesting to use client-side. > > > > Adding a client configuration property for clients may be an overkill > > here. What I am going to ask is naive but - is it theoretically possible > > for the broker to legitimately send responses over 100 MB in size? > > > > Thanks, > > Alexandre > > > > [1] https://github.com/apache/kafka/pull/5940 > > [2] > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, fatal error: 80: problem > > unwrapping net record > > > > javax.net.ssl.SSLException: Unrecognized SSL message, plaintext > connection? > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, SEND TLSv1.2 ALERT: > fatal, > > description = internal_error > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, WRITE: TLSv1.2 Alert, > > length = 2 > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, called closeOutbound() > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, closeOutboundInternal() > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, called closeInbound() > > > > kafka-network-thread-0-ListenerName(SSL)-SSL-4, fatal: engine already > > closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before > > receiving peer's close_notify: possible truncation attack? > > > > [Raw write]: length = 7 > > > > : 15 03 03 00 02 02 50 ..P > > > > > > Le jeu. 1 août 2019 à 08:50, Stanislav Kozlovski > > > a écrit : > > > >> Hey Alexandre, thanks for the KIP! > >> > >> I had personally hit the same problem and found it very annoying. > >> Have you considered detecting such a scenario in the client and simply > >> throwing an exception, rather than allocating any memory? > >> I have an open PR that does just that - > >> https://github.com/apache/kafka/pull/5940 > >> <https://github.com/apache/kafka/pull/5940/files> > >> > >> Best, > >> Stanislav > >> > >> On Wed, Jul 31, 2019 at 10:35 AM Gokul Ramanan Subramanian < > >> gokul24...@gmail.com> wrote: > >> > >> > Hi Alex. > >> > > >> > A rejected alternative is to try to do SSL handshake from the > plaintext > >> > transport layer. This couples various transport layers and is > >> inflexible to > >> > adding new transport layers in the future. Further, if the plaintext > >> > transport layer does SSL handshake, it will always get an error, > >> > irrespective of whether or not the peer supports SSL. Therefore, the > >> > plaintext transport layer would have to determine if the peer uses SSL > >> or > >> > not based on the type of error it gets back from the SSL handshake. > This > >> > fits right into the general anti-pattern of using exceptions as > control > >> > flow mechanisms. > >> > > >> > Another rejected alternative would be to have a single byte at the > >> > transport layer represent the security protocol in use. This byte > would > >> > have to be present in every single message between clients and brokers > >> and > >> > between brokers and brokers. This change is backwards-incompatible and > >> adds > >> > overhead to Kafka. It is likely never going to get accepted by the > >> > community. > >> > > >> > In the absence of a fool-proof way to do a handhskake across all > >> security > >> > protocols (plaintext, SSL, other future ones), I think that the > proposed > >> > KIP provides a good solution. Therefore, you have my +1. (Not sure if > >> my +1 > >> > counts, since I am a Kafka noob). > >> > > >> > Thanks. > >> > Gokul Ramanan Subramanian > >> > Senior SDE, Amazon AWS > >> > > >> > On 28/Jul/19 05:43:19PM +0100, Alexandre Dupriez wrote: > >> > Hello, > >> > > >> > I have created the KIP-498 > >> > < > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM > >> > > > >> > to > >> > propose a minor addition to the producer configuration properties, in > >> > order > >> > to protect against OOM when the response message decoded by a client > >> > indicates an "abnormally high" size to be allocated. > >> > > >> > This follows this discussion thread > >> > <https://www.mail-archive.com/dev@kafka.apache.org/msg55658.html> > and > >> is > >> > tracked by KAFKA-4090 < > >> https://issues.apache.org/jira/browse/KAFKA-4090>. > >> > > >> > Please let me know what you think. I created this KIP even though the > >> > change seems minimal because it affects client configuration, which > is > >> > mentioned as a type of change eligible for a KIP on this main wiki > page > >> > < > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > >> > > > >> > . > >> > > >> > Thanks, > >> > Alexandre > >> > > >> > > > -- GOKUL R. SUBRAMANIAN, Software Engineer
Re: [DISCUSS] KIP-498: Add client-side configuration for maximum response size to protect against OOM
Hi Alex. A rejected alternative is to try to do SSL handshake from the plaintext transport layer. This couples various transport layers and is inflexible to adding new transport layers in the future. Further, if the plaintext transport layer does SSL handshake, it will always get an error, irrespective of whether or not the peer supports SSL. Therefore, the plaintext transport layer would have to determine if the peer uses SSL or not based on the type of error it gets back from the SSL handshake. This fits right into the general anti-pattern of using exceptions as control flow mechanisms. Another rejected alternative would be to have a single byte at the transport layer represent the security protocol in use. This byte would have to be present in every single message between clients and brokers and between brokers and brokers. This change is backwards-incompatible and adds overhead to Kafka. It is likely never going to get accepted by the community. In the absence of a fool-proof way to do a handhskake across all security protocols (plaintext, SSL, other future ones), I think that the proposed KIP provides a good solution. Therefore, you have my +1. (Not sure if my +1 counts, since I am a Kafka noob). Thanks. Gokul Ramanan Subramanian Senior SDE, Amazon AWS On 28/Jul/19 05:43:19PM +0100, Alexandre Dupriez wrote: Hello, I have created the KIP-498 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM> to propose a minor addition to the producer configuration properties, in order to protect against OOM when the response message decoded by a client indicates an "abnormally high" size to be allocated. This follows this discussion thread <https://www.mail-archive.com/dev@kafka.apache.org/msg55658.html> and is tracked by KAFKA-4090 <https://issues.apache.org/jira/browse/KAFKA-4090>. Please let me know what you think. I created this KIP even though the change seems minimal because it affects client configuration, which is mentioned as a type of change eligible for a KIP on this main wiki page <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals> . Thanks, Alexandre
JIRA and KIP contributor permissions
Hello Community, In order to start contributing to Apache Kafka project, could I please request contributor access to JIRA and be granted write permissions to the Kafka wiki? My name: Gokul Ramanan Subramanian JIRA username: gokul2411s Confluence username: gokul2411s (https://cwiki.apache.org/confluence/display/~gokul2411s) Committer email: gokus...@amazon.com Thank you in advance. Gokul Ramanan Subramanian
