Thanks for the comments Jay and Todd. Firstly, I'd like to address some of
Jay's points.
1. You are right that we don't need to disable quotas on a per-client basis.
2. Configuration management: I must admit, I haven't read Joe's proposal in
detail. My thinking was that we can keep configuration management separate from
this discussion since this is already quite a meaty topic. Let me spend some
time reading that KIP and then I can add more detail to the quota KIP.
3. Custom Quota implementations: I don't think it is necessarily a bad idea to
have a interface called the QuotaManager(RequestThrottler). This doesn't
necessarily mean exposing the interface as a public API. It is a mechanism to
limit code changes to 1-2 specific classes. It prevents quota logic from
bleeding into multiples places in the code as happens in any big piece of code.
I fully agree that we should not expose this as a public API unless there is a
very strong reason to. This seems to be more of an implementation detail.
4. Metrics Package: I'll add a section on the wiki about using things from the
metrics package. Currently, the quota stuff is located in
"clients/common/metrics". This means that we will have to migrate all that
functionality into core. Do this also mean that we will need to replace the
existing metrics code in "core" with the newly imported package as a part of
this project? If so, that's a relatively large undertaking and it needs to be
discussed separately IMO.
5. Request Throttler vs QuotaManager -
I wanted my quota manager to do something similar to what you proposed. Inside
KafkaApis, I could do:
if(quotaManager.check())
// process request
else
return
Internally QuotaManager:check() could do exactly what you suggested
try {
quotaMetric.record(newVal)
} catch (QuotaException e) {
// logic to calculate delay
requestThrottler.add(new DelayedResponse(...), ...)
return
}
This approach gives us the flexibility of deciding what metric we want to
record inside QuotaManager. This brings us back to the same discussion of
pluggable quota policies. It's a bit hard to articulate, but for example: this
allows us to quota on bytes in/out vs total number of requests in/out. It also
encapsulates the logic to calculate delay. Recording the metrics in KafkaApis
would make it look more complex.
try {
bytesPerSecondMetrics.record(newVal);
produceRequestsPerSecondMetrics.record(newVal);
}
catch(QuotaException) {
// logic to calculate delay
requestThrottler.add(new DelayedResponse(...), ...)
}
Stepping back a bit, I see that this is all internal to KafkaApis and we have
the flexibility of refactoring this as we see fit at a later date. It should be
ok to not have RequestThrottler, QuotaManager be an interface but proper
implementations instead.
6. As for assigning exact delays to requests, I agree that having a 30 second
delay is rather bad. I was thinking that we could have 5 second windows
instead. Even if a client exhausts their quota in the first 2 seconds, they
will only be delayed for another 3 seconds. Is there a reason this isn't good
enough? I'm also going to dig into the existing quota package and see what we
can leverage.
7. Exposing quota usage: We do plan to expose the quota metrics via JMX. Adding
an API to query quota status is absolutely feasible and also a good idea.
Thanks,
Aditya
________________________________________
From: Jay Kreps [jay.kr...@gmail.com]
Sent: Monday, March 09, 2015 5:01 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
Hey Todd,
Nice points, let me try to respond:
Plugins
Yeah let me explain what I mean about plugins. The contracts that matter
for us are public contracts, i.e. the protocol, the binary format, stuff in
zk, and the various plug-in apis we expose. Adding an internal interface
later will not be hard--the quota check is going to be done in 2-6 places
in the code which would need to be updated, all internal to the broker.
The challenge with making things pluggable up front is that the policy is
usually fairly trivial to plug in but each policy requires different
inputs--the number of partitions, different metrics, etc. Once we give a
public api it is hard to add to it without breaking the original contract
and hence breaking everyones plugins. So if we do this we want to get it
right early if possible. In any case I think whether we want to design a
pluggable api or just improve a single implementation, the work we need to
do is the same: brainstorm the set of use cases the feature has and then
figure out the gap in our proposed implementation that leaves some use case
uncovered. Once we have these specific cases we can try to figure out if
that could be solved with a plugin or by improving our default proposal.
Enforcement
I started out arguing your side (immediate error), but I have switched to
preferring delay. Here is what convinced me, let's see if it moves you.
First, the delay quota need not hold onto any request data. The produce
response can be delayed after the request is completed and the fetch can be
delayed prior to the fetch being executed. So no state needs to be
maintained in memory, other than a single per-connection token. This is a
really important implementation detail for large scale usage that I didn't
realize at first. I would agree that maintaining a request per connection
in memory is a non-starter for an environment with 10s of thousands of
connections.
The second argument is that I think this really expands the use cases where
the quotas can be applicable.
The use case I have heard people talk about is event collection from apps.
In this use case the app is directly sending data at a more or less steady
state and never really has a performance spike unless the app has a bug or
the application itself experiences more traffic. So in this case you should
actually never hit the quota, and if you do, the data is going to be
dropped wither it is dropped by the server with an error or by the client.
These use cases will never block the app (which would be dangerous) since
the client is always non-blocking and drops data when it's buffer is full
rather than blocking. I agree that for this use case either server-side
delay or client side delay are both reasonable--the pro of a server-side
delay is that it doesn't require special client handling, the pro of the
server-side error is that it is more transparent.
But now consider non-steady-state use cases. Here I am thinking of:
1. Data load from Hadoop
2. MM load into a cluster with live usage
3. Database changelog capture
4. Samza
Each of these has a case where it is "catching up" or otherwise slammed by
load from the source system:
1. A M/R job dump a ton of data all at once
2. MM when catching up after some downtime
3. Database changelog will have a load phase when populating data for the
first time
4. Samza when restoring state or catching up after fail-over
In each of these cases you want the consumer or producer to go as fast as
possible but not impact the other users of the cluster. In these cases you
are actually using the quotas totally differently. In the app event capture
use case the quota was more like a safety valve that you expected to never
hit. However in these cases I just listed you fully expect to hit and
remain at the quota for extended periods of time and that will be totally
normal.
These are the cases where throttling throughput is better than sending an
error. If we send an error then any producer who doesn't configure enough
retries is going to start losing data. Further even if you set infinite
retries the retry itself is going to mean resending the data over and over
until you get a non-error. This is bad because in a period of high load you
are then going to be incurring more network load as lots of producers start
retrying (this isn't a problem on the consumer because the fetch request is
small but is an issue on the producer).
I take your point about the potential danger of slowing down a producing
app that is configured to block. But actually this danger is no different
than what will happen if it exceeds the node capacity now--when that
happens requests will start getting slow and the app will block. The only
difference is that that limit is now lower than when the node's capacity is
totally exhausted. So I don't think that is a new danger.
Exposing quota usage
I agree we should make this available, good use of this feature obviously
means knowing how close you are to your quota before you hit it.
Cheers,
-Jay
On Mon, Mar 9, 2015 at 10:34 AM, Todd Palino <tpal...@gmail.com> wrote:
> First, a couple notes on this...
>
> 3 - I generally agree with the direction of not pre-optimizing. However, in
> this case I'm concerned about the calculation of the cost of doing plugins
> now vs. trying to refactor the code to do it later. It would seem to me
> that doing it up front will have less friction. If we wait to do plugins
> later, it will probably mean changing a lot of related code which will be
> significantly more work. We've spent a lot of time talking about various
> implementations, and I think it not unreasonable to believe that what one
> group wants initially is not going to solve even most cases, as it will
> vary by use case.
>
> 4 - I really disagree with this. Slowing down a request means that you're
> going to hold onto it in the broker. This takes up resources and time, and
> is generally not the way other services handle quota violations. In
> addition you are causing potential problems with the clients by taking a
> call that's supposed to return as quickly as possible and making it take a
> long time. This increases latency and deprives the client of the ability to
> make good decisions about what to do. By sending an error back to the
> client you inform them of what the problem is, and you allow the client to
> make an intelligent decision, such as queuing to send later, sending to
> another resource, or handling anything from their upstreams differently.
>
> You're absolutely right that throwing back an immediate error has the
> potential to turn a quota violation into a different problem for a badly
> behaved client. But OS and upstream networking tools can see a problem
> based on a layer 4 issue (rapidly reconnecting client) rather than layers
> above. Out of the options provided, I think A is the correct choice. B
> seems to be the most work (you have the delay, and the client still has to
> handle errors and backoff), and C is what I disagree with doing.
>
> I would also like to see a provision for allowing the client to query its
> quota status within the protocol. I think we should allow for a request (or
> information within an existing response) where the client can ask what its
> current quota status is. This will allow for the clients to manage their
> quotas, and it will allow for emitting metrics on the client side for quota
> status (rather than relying on the server-side metrics, which tends to put
> the responsibility in the wrong place).
>
>
> -Todd
>
>
> On Sun, Mar 8, 2015 at 10:52 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
>
> > Hey Adi,
> >
> > Great write-up. Here are some comments:
> >
> > 1. I don't think you need a way to disable quotas on a per-client basis,
> > that is just the equivalent of setting the quota to be infinite, right?
> >
> > 2. I agree that the configuration problem is a general part of doing
> > dynamic configuration, and it is right to separate that into the config
> > KIP. But Joe's proposal currently doesn't provide nearly what you need in
> > its current form--it doesn't even handle client-id based configuration,
> let
> > alone the notification mechanism you would need to update your quota--so
> we
> > really need to give completely explicitly how that KIP is going to solve
> > this problem.
> >
> > 3. Custom quota implementations: let's do this later. Pluggability comes
> > with a high cost and we want to try really hard to avoid it. So in the
> > future if we have a really solid case for an alternative quota approach
> > let's see if we can't improve the current approach and stick with one
> good
> > implementation. If we really can't then let's add a plugin system. I
> think
> > doing it now is premature.
> >
> > 4. I think the ideal quota action from the users point of view is just to
> > slow down the writer or reader transparently to match their capacity
> > allocation. Let's try to see if we can make that work.
> >
> > I think immediate error can be ruled out entirely because it depends on
> the
> > client properly backing off. In cases where they don't we may actually
> make
> > things worse. Given the diversity of clients I think this is probably not
> > going to happen.
> >
> > The only downside to just delaying the request that was pointed out was
> > that if the delay exceeded the request timeout the user might retry. This
> > is true but it applies to any approach that delays requests (both B and
> C).
> > I think with any sane request timeout and quota the per request delay we
> > induce will be way lower (otherwise you would be hitting the timeout all
> > the time just due to linux I/O variance, in which case you can't really
> > complain).
> >
> > 5. We need to explain the relationship between the quota stuff in the
> > metrics package and this. We need to either remove that stuff or use it.
> We
> > can't have two quota things. Since quota fundamentally apply to windowed
> > metrics, I would suggest doing whatever improvements to that to make it
> > usable for quotas.
> >
> > 6. I don't think the quota manager interface is really what we need if
> I'm
> > understanding it correctly. You give a method
> > <T extends RequestOrResponse> boolean check(T request);
> > But how would you implement this method? It seems like it would basically
> > internally just be a switch statement with a different check for each
> > request type. So this is a pretty terrible object oriented api, right? It
> > seems like what we will be doing is taking code that would otherwise just
> > be in the request handling flow, and moving it into this method, with a
> > bunch of instanceof checks?
> >
> > I think what we need is just a delayqueue and a background thread that
> > sends the delayed responses (we were calling it a purgatory but it isn't,
> > it is just a timeout based delay--there are no watchers or keys or any of
> > that).
> >
> > Let's rename the QuotaManager RequestThrottler and have it just have a
> > single method:
> > class RequestThrottler {
> > sendDelayedResponse(response, delay, timeunit)
> > }
> > internally it will put the response into the delay queue and there will
> be
> > a background thread that sends out those responses after the delay
> elapses.
> >
> > So usage in KafkaApis would look like:
> > try {
> > quotaMetric.record(newVal)
> > } catch (QuotaException e) {
> > requestThrottler.add(new DelayedResponse(...), ...)
> > return
> > }
> >
> > The advantage of this is that the logic of what metric is being checked
> and
> > the logic of how to appropriately correct the response, both of which
> will
> > be specific to each request, now remain in KafkaApis where they belong.
> The
> > throttler just delays the sending of the response for the appropriate
> time
> > and has no per-request logic whatsoever.
> >
> > 7. We need to think through and state the exact algorithm for how we will
> > assign delays to requests for a use case that is over its quota. That is
> > closely tied to how we calculate the metric used. Here would be a bad
> > approach we should not use:
> > a. measure in a 30 second window.
> > b. when we have hit the cap in that window, delay for the remainder of
> the
> > 30 seconds
> > As you can imagine with this bad algorithm you might then use all server
> > resources for 5 seconds, then suddenly assign a 25 second delay to the
> next
> > request from that client, then the window would reset and this would
> > repeat.
> > The quota package is already doing a good job of the windowed metrics,
> but
> > we'll want to integrate the backoff calculation with that algorithm
> > (assuming that is what we are using).
> >
> > Cheers,
> >
> > -Jay
> >
> > On Wed, Mar 4, 2015 at 3:51 PM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Posted a KIP for quotas in kafka.
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > >
> > > Appreciate any feedback.
> > >
> > > Aditya
> > >
> >
>
