Hey Becket/Rajini, When I thought about it more deeply I came around to the "percent of processing time" metric too. It seems a lot closer to the thing we actually care about and need to protect. I also think this would be a very useful metric even in the absence of throttling just to debug whose using capacity.
Two problems to consider: 1. I agree that for the user it is understandable what lead to their being throttled, but it is a bit hard to figure out the safe range for them. i.e. if I have a new app that will send 200 messages/sec I can probably reason that I'll be under the throttling limit of 300 req/sec. However if I need to be under a 10% CPU resources limit it may be a bit harder for me to know a priori if i will or won't. 2. Calculating the available CPU time is a bit difficult since there are actually two thread pools--the I/O threads and the network threads. I think it might be workable to count just the I/O thread time as in the proposal, but the network thread work is actually non-trivial (e.g. all the disk reads for fetches happen in that thread). If you count both the network and I/O threads it can skew things a bit. E.g. say you have 50 network threads, 10 I/O threads, and 8 cores, what is the available cpu time available in a second? I suppose this is a problem whenever you have a bottleneck between I/O and network threads or if you end up significantly over-provisioning one pool (both of which are hard to avoid). An alternative for CPU throttling would be to use this api: http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/ThreadMXBean.html#getThreadCpuTime(long) That would let you track actual CPU usage across the network, I/O threads, and purgatory threads and look at it as a percentage of total cores. I think this fixes many problems in the reliability of the metric. It's meaning is slightly different as it is just CPU (you don't get charged for time blocking on I/O) but that may be okay because we already have a throttle on I/O. The downside is I think it is possible this api can be disabled or isn't always available and it may also be expensive (also I've never used it so not sure if it really works the way i think). -Jay On Mon, Feb 20, 2017 at 3:17 PM, Becket Qin <becket....@gmail.com> wrote: > If the purpose of the KIP is only to protect the cluster from being > overwhelmed by crazy clients and is not intended to address resource > allocation problem among the clients, I am wondering if using request > handling time quota (CPU time quota) is a better option. Here are the > reasons: > > 1. request handling time quota has better protection. Say we have request > rate quota and set that to some value like 100 requests/sec, it is possible > that some of the requests are very expensive actually take a lot of time to > handle. In that case a few clients may still occupy a lot of CPU time even > the request rate is low. Arguably we can carefully set request rate quota > for each request and client id combination, but it could still be tricky to > get it right for everyone. > > If we use the request time handling quota, we can simply say no clients can > take up to more than 30% of the total request handling capacity (measured > by time), regardless of the difference among different requests or what is > the client doing. In this case maybe we can quota all the requests if we > want to. > > 2. The main benefit of using request rate limit is that it seems more > intuitive. It is true that it is probably easier to explain to the user > what does that mean. However, in practice it looks the impact of request > rate quota is not more quantifiable than the request handling time quota. > Unlike the byte rate quota, it is still difficult to give a number about > impact of throughput or latency when a request rate quota is hit. So it is > not better than the request handling time quota. In fact I feel it is > clearer to tell user that "you are limited because you have taken 30% of > the CPU time on the broker" than otherwise something like "your request > rate quota on metadata request has reached". > > Thanks, > > Jiangjie (Becket) Qin > > > On Mon, Feb 20, 2017 at 2:23 PM, Jay Kreps <j...@confluent.io> wrote: > > > I think this proposal makes a lot of sense (especially now that it is > > oriented around request rate) and fills the biggest remaining gap in the > > multi-tenancy story. > > > > I think for intra-cluster communication (StopReplica, etc) we could avoid > > throttling entirely. You can secure or otherwise lock-down the cluster > > communication to avoid any unauthorized external party from trying to > > initiate these requests. As a result we are as likely to cause problems > as > > solve them by throttling these, right? > > > > I'm not so sure that we should exempt the consumer requests such as > > heartbeat. It's true that if we throttle an app's heartbeat requests it > may > > cause it to fall out of its consumer group. However if we don't throttle > it > > it may DDOS the cluster if the heartbeat interval is set incorrectly or > if > > some client in some language has a bug. I think the policy with this kind > > of throttling is to protect the cluster above any individual app, right? > I > > think in general this should be okay since for most deployments this > > setting is meant as more of a safety valve---that is rather than set > > something very close to what you expect to need (say 2 req/sec or > whatever) > > you would have something quite high (like 100 req/sec) with this meant to > > prevent a client gone crazy. I think when used this way allowing those to > > be throttled would actually provide meaningful protection. > > > > -Jay > > > > > > > > On Fri, Feb 17, 2017 at 9:05 AM, Rajini Sivaram <rajinisiva...@gmail.com > > > > wrote: > > > > > Hi all, > > > > > > I have just created KIP-124 to introduce request rate quotas to Kafka: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 124+-+Request+rate+quotas > > > > > > The proposal is for a simple percentage request handling time quota > that > > > can be allocated to *<client-id>*, *<user>* or *<user, client-id>*. > There > > > are a few other suggestions also under "Rejected alternatives". > Feedback > > > and suggestions are welcome. > > > > > > Thank you... > > > > > > Regards, > > > > > > Rajini > > > > > >
