RE: [KIP-DISCUSSION] KIP-13 Quotas

[Aditya Auradkar]

Thanks for the review Guozhang.

1. Agreed. 

2. This proposal actually waits for the maximum of the 2 timeouts. This reduces 
implementation complexity at the cost of waiting longer than strictly needed 
for quotas. Note that this is only for the case where acks=-1. 

However we can solve this if it is a significant concern by adding watcher keys 
for all partitions (only if acks=-1). These are the keys we would normally add 
while waiting for acknowledgements. We can change the tryComplete() function to 
return false until 'quota_timeout' time has elapsed AND all the 
acknowledgements have been received.

Thanks,
Aditya
________________________________________
From: Guozhang Wang [wangg...@gmail.com]
Sent: Wednesday, April 15, 2015 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the summary. A few comments below:

1. Say a produce request has replication timeout X, and upon finishing the
local append it is determined to be throttled Y ms where Y > X, then after
it has timed out in the purgatory after Y ms we should still check if the
#.acks has fulfilled in order to set the correct error codes in the
response.

2. I think it is actually common that the calculated throttle time Y is
less than the replication timeout X, which will be a tricky case since we
need to make sure 1) at least the request it held in the purgatory for Y
ms, 2) after Y ms elapsed, if the #.acks has fulfilled within X ms then set
no-error-code and return immediately, 3) after X ms elapsed, set
timeout-error-code and return.

Guozhang

On Tue, Apr 14, 2015 at 5:01 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> This is an implementation proposal for delaying requests in quotas using
> the current purgatory. I'll discuss the usage for produce and fetch
> requests separately.
>
> 1. Delayed Produce Requests - Here, the proposal is basically to reuse
> DelayedProduce objects and insert them into the purgatory with no watcher
> keys if the request is being throttled. The timeout used in the request
> should be the Max(quota_delay_time, replication_timeout).
> In most cases, the quota timeout should be greater than the existing
> timeout but in order to be safe, we can use the maximum of these values.
> Having no watch keys will allow the operation to be enqueued directly into
> the timer and will not add any overhead in terms of watching keys (which
> was a concern). In this case, having watch keys is not beneficial since the
> operation must be delayed for a fixed amount of time and there is no
> possibility for the operation to complete before the timeout i.e.
> tryComplete() can never return true before the timeout. On timeout, since
> the operation is a TimerTask, the timer will call run() which calls
> onComplete().
> In onComplete, the DelayedProduce can repeat the check in tryComplete()
> (only if acks=-1 whether all replicas fetched upto a certain offset) and
> return the response immediately.
>
> Code will be structured as follows in ReplicaManager:appendMessages()
>
> if(isThrottled) {
>   fetch = new DelayedProduce(timeout)
>   purgatory.tryCompleteElseWatch(fetch, Seq())
> }
> else if(delayedRequestRequired()) {
>  // Insert into purgatory with watched keys for unthrottled requests
> }
>
> In this proposal, we avoid adding unnecessary watches because there is no
> possibility of early completion and this avoids any potential performance
> penalties we were concerned about earlier.
>
> 2. Delayed Fetch Requests - Similarly, the proposal here is to reuse the
> DelayedFetch objects and insert them into the purgatory with no watcher
> keys if the request is throttled. Timeout used is the Max(quota_delay_time,
> max_wait_timeout). Having no watch keys provides the same benefits as
> described above. Upon timeout, the onComplete() is called and the operation
> proceeds normally i.e. perform a readFromLocalLog and return a response.
> The caveat here is that if the request is throttled but the throttle time
> is less than the max_wait timeout on the fetch request, the request will be
> delayed to a Max(quota_delay_time, max_wait_timeout). This may be more than
> strictly necessary (since we are not watching for
> satisfaction on any keys).
>
> I added some testcases to DelayedOperationTest to verify that it is
> possible to schedule operations with no watcher keys. By inserting elements
> with no watch keys, the purgatory simply becomes a delay queue. It may also
> make sense to add a new API to the purgatory called
> delayFor() that basically accepts an operation without any watch keys
> (Thanks for the suggestion Joel).
>
> Thoughts?
>
> Thanks,
> Aditya
>
> ________________________________________
> From: Guozhang Wang [wangg...@gmail.com]
> Sent: Monday, April 13, 2015 7:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> I think KAFKA-2063 (bounding fetch response) is still under discussion, and
> may not be got it in time with KAFKA-1927.
>
> On Thu, Apr 9, 2015 at 4:49 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > I think it's reasonable to batch the protocol changes together. In
> > addition to the protocol changes, is someone actively driving the server
> > side changes/KIP process for KAFKA-2063?
> >
> > Thanks,
> > Aditya
> >
> > ________________________________________
> > From: Jun Rao [j...@confluent.io]
> > Sent: Thursday, April 09, 2015 8:59 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> >
> > Since we are also thinking about evolving the fetch request protocol in
> > KAFKA-2063 (bound fetch response size), perhaps it's worth thinking
> through
> > if we can just evolve the protocol once.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Apr 8, 2015 at 10:43 AM, Aditya Auradkar <
> > aaurad...@linkedin.com.invalid> wrote:
> >
> > > Thanks for the detailed review. I've addressed your comments.
> > >
> > > For rejected alternatives, we've rejected per-partition distribution
> > > because we choose client based quotas where there is no notion of
> > > partitions. I've explained in a bit more detail in that section.
> > >
> > > Aditya
> > >
> > > ________________________________________
> > > From: Joel Koshy [jjkosh...@gmail.com]
> > > Sent: Wednesday, April 08, 2015 6:30 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > >
> > > Thanks for updating the wiki. Looks great overall. Just a couple
> > > more comments:
> > >
> > > Client status code:
> > > - v0 requests -> current version (0) of those requests.
> > > - Fetch response has a throttled flag instead of throttle time -  I
> > >   think you intended the latter.
> > > - Can you make it clear that the quota status is a new field
> > >   called throttleTimeMs (or equivalent). It would help if some of
> > >   that is moved (or repeated) in compatibility/migration plan.
> > > - So you would need to upgrade brokers first, then the clients.
> > >   While upgrading the brokers (via a rolling bounce) the brokers
> > >   cannot start using the latest fetch-request version immediately
> > >   (for replica fetches). Since there will be older brokers in the mix
> > >   those brokers would not be able to read v1 fetch requests. So all
> > >   the brokers should be upgraded before switching to the latest
> > >   fetch request version. This is similar to what Gwen proposed in
> > >   KIP-2/KAFKA-1809 and I think we will need to use the
> > >   inter-broker protocol version config.
> > >
> > > Rejected alternatives-quota-distribution.B: notes that this is the
> > > most elegant model, but does not explain why it was rejected. I
> > > think this was because we would then need some sort of gossip
> > > between brokers since partitions are across the cluster. Can you
> > > confirm?
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Wed, Apr 08, 2015 at 05:45:34AM +0000, Aditya Auradkar wrote:
> > > > Hey everyone,
> > > >
> > > > Following up after today's hangout. After discussing the client side
> > > metrics piece internally, we've incorporated that section into the KIP.
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > >
> > > > Since there appears to be sufficient consensus, I'm going to start a
> > > voting thread.
> > > >
> > > > Thanks,
> > > > Aditya
> > > > ________________________________________
> > > > From: Gwen Shapira [gshap...@cloudera.com]
> > > > Sent: Tuesday, April 07, 2015 11:31 AM
> > > > To: Sriharsha Chintalapani
> > > > Cc: dev@kafka.apache.org
> > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > >
> > > > Yeah, I was not suggesting adding auth to metrics - I think this
> > > needlessly
> > > > complicates everything.
> > > > But we need to assume that client developers will not have access to
> > the
> > > > broker metrics (because in secure environment they probably won't).
> > > >
> > > > Gwen
> > > >
> > > > On Tue, Apr 7, 2015 at 11:20 AM, Sriharsha Chintalapani <
> > ka...@harsha.io
> > > >
> > > > wrote:
> > > >
> > > > > Having auth  on top of metrics is going to be lot more difficult.
> How
> > > are
> > > > > we going to restrict metrics reporter which run as part of kafka
> > server
> > > > > they will have access to all the metrics and they can publish to
> > > ganglia
> > > > > etc..  I look at the metrics as a read-only info. As you said
> metrics
> > > for
> > > > > all the topics can be visible but what actions are we looking that
> > can
> > > be
> > > > > non-secure based on metrics alone? . This probably can be part of
> > > KIP-11
> > > > > discussion.
> > > > >  Having said that it will be great if the throttling details can be
> > > > > exposed as part of the response to the client. Instead of looking
> at
> > > > > metrics , client can depend on the response to slow down if its
> being
> > > > > throttled.  This allows us the clients can be self-reliant based on
> > the
> > > > > response .
> > > > >
> > > > > --
> > > > > Harsha
> > > > >
> > > > >
> > > > > On April 7, 2015 at 9:55:41 AM, Gwen Shapira (
> gshap...@cloudera.com)
> > > > > wrote:
> > > > >
> > > > > Re (1):
> > > > > We have no authorization story on the metrics collected by brokers,
> > so
> > > I
> > > > > assume that access to broker metrics means knowing exactly which
> > topics
> > > > > exist and their throughputs. (Prath and Don, correct me if I got it
> > > > > wrong...)
> > > > > Secure environments will strictly control access to this
> information,
> > > so I
> > > > > am pretty sure the client developers will not have access to server
> > > > > metrics
> > > > > at all.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Tue, Apr 7, 2015 at 7:41 AM, Jay Kreps <jay.kr...@gmail.com>
> > wrote:
> > > > >
> > > > > > Totally. But is that the only use? What I wanted to flesh out was
> > > > > whether
> > > > > > the goal was:
> > > > > > 1. Expose throttling in the client metrics
> > > > > > 2. Enable programmatic response (i.e. stop sending stuff or
> > something
> > > > > like
> > > > > > that)
> > > > > >
> > > > > > I think I kind of understand (1) but let's get specific on the
> > > metric we
> > > > > > would be adding and what exactly you would expose in a dashboard.
> > For
> > > > > > example if the goal is just monitoring do I really want a boolean
> > > flag
> > > > > for
> > > > > > is_throttled or do I want to know how much I am being throttled
> > (i.e.
> > > > > > throttle_pct might indicate the percent of your request time that
> > was
> > > > > due
> > > > > > to throttling or something like that)? If I am 1% throttled that
> > may
> > > be
> > > > > > irrelevant but 99% throttled would be quite relevant? Not sure I
> > > agree,
> > > > > > just throwing that out there...
> > > > > >
> > > > > > For (2) the prior discussion seemed to kind of allude to this
> but I
> > > > > can't
> > > > > > really come up with a use case. Is there one?
> > > > > >
> > > > > > If it is just (1) I think the question is whether it really helps
> > > much
> > > > > to
> > > > > > have the metric on the client vs the server. I suppose this is a
> > bit
> > > > > > environment specific. If you have a central metrics system it
> > > shouldn't
> > > > > > make any difference, but if you don't I suppose it does.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira <
> > gshap...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Here's a wild guess:
> > > > > > >
> > > > > > > An app developer included a Kafka Producer in his app, and is
> not
> > > > > happy
> > > > > > > with the throughput. He doesn't have visibility into the
> brokers
> > > since
> > > > > > they
> > > > > > > are owned by a different team. Obviously the first instinct of
> a
> > > > > > developer
> > > > > > > who knows that throttling exists is to blame throttling for any
> > > > > slowdown
> > > > > > in
> > > > > > > the app.
> > > > > > > If he doesn't have a way to know from the responses whether or
> > not
> > > his
> > > > > > app
> > > > > > > is throttled, he may end up calling Aditya at 4am asked "Hey,
> is
> > my
> > > > > app
> > > > > > > throttled?".
> > > > > > >
> > > > > > > I assume Aditya is trying to avoid this scenario.
> > > > > > >
> > > > > > > On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps <jay.kr...@gmail.com
> >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hey Aditya,
> > > > > > > >
> > > > > > > > 2. I kind of buy it, but I really like to understand the
> > details
> > > of
> > > > > the
> > > > > > > use
> > > > > > > > case before we make protocol changes. What changes are you
> > > proposing
> > > > > in
> > > > > > > the
> > > > > > > > clients for monitoring and how would that be used?
> > > > > > > >
> > > > > > > > -Jay
> > > > > > > >
> > > > > > > > On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar <
> > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Jay,
> > > > > > > > >
> > > > > > > > > 2. At this time, the proposed response format changes are
> > only
> > > for
> > > > > > > > > monitoring/informing clients. As Jun mentioned, we get
> > instance
> > > > > level
> > > > > > > > > monitoring in this case since each instance that got
> > throttled
> > > > > will
> > > > > > > have
> > > > > > > > a
> > > > > > > > > metric confirming the same. Without client level monitoring
> > for
> > > > > this,
> > > > > > > > it's
> > > > > > > > > hard for application developers to find if they are being
> > > > > throttled
> > > > > > > since
> > > > > > > > > they will also have to be aware of all the brokers in the
> > > cluster.
> > > > > > This
> > > > > > > > is
> > > > > > > > > quite problematic for large clusters.
> > > > > > > > >
> > > > > > > > > It seems nice for app developers to not have to think about
> > > kafka
> > > > > > > > internal
> > > > > > > > > metrics and only focus on the metrics exposed on their
> > > instances.
> > > > > > > > Analogous
> > > > > > > > > to having client-sde request latency metrics. Basically, we
> > > want
> > > > > an
> > > > > > > easy
> > > > > > > > > way for clients to be aware if they are being throttled.
> > > > > > > > >
> > > > > > > > > 4. For purgatory v delay queue, I think we are on the same
> > > page. I
> > > > > > feel
> > > > > > > > it
> > > > > > > > > is nicer to use the purgatory but I'm happy to use a
> > > DelayQueue if
> > > > > > > there
> > > > > > > > > are performance implications. I don't know enough about the
> > > > > current
> > > > > > and
> > > > > > > > > Yasuhiro's new implementation to be sure one way or the
> > other.
> > > > > > > > >
> > > > > > > > > Stepping back, I think these two things are the only
> > remaining
> > > > > point
> > > > > > of
> > > > > > > > > discussion within the current proposal. Any concerns if I
> > > started
> > > > > a
> > > > > > > > voting
> > > > > > > > > thread on the proposal after the KIP discussion tomorrow?
> > > > > (assuming
> > > > > > we
> > > > > > > > > reach consensus on these items)
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aditya
> > > > > > > > > ________________________________________
> > > > > > > > > From: Jay Kreps [jay.kr...@gmail.com]
> > > > > > > > > Sent: Saturday, April 04, 2015 1:36 PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > >
> > > > > > > > > Hey Aditya,
> > > > > > > > >
> > > > > > > > > 2. For the return flag I'm not terribly particular. If we
> > want
> > > to
> > > > > add
> > > > > > > it
> > > > > > > > > let's fully think through how it will be used. The only
> > > concern I
> > > > > > have
> > > > > > > is
> > > > > > > > > adding to the protocol without really thinking through the
> > use
> > > > > cases.
> > > > > > > So
> > > > > > > > > let's work out the APIs we want to add to the Java consumer
> > and
> > > > > > > producer
> > > > > > > > > and the use cases for how clients will make use of these.
> For
> > > my
> > > > > > part I
> > > > > > > > > actually don't see much use other than monitoring since it
> > > isn't
> > > > > an
> > > > > > > error
> > > > > > > > > condition to be at your quota. And if it is just
> monitoring I
> > > > > don't
> > > > > > > see a
> > > > > > > > > big enough difference between having the monitoring on the
> > > > > > server-side
> > > > > > > > > versus in the clients to justify putting it in the
> protocol.
> > > But I
> > > > > > > think
> > > > > > > > > you guys may have other use cases in mind of how a client
> > would
> > > > > make
> > > > > > > some
> > > > > > > > > use of this? Let's work that out. I also don't feel
> strongly
> > > about
> > > > > > > it--it
> > > > > > > > > wouldn't be *bad* to have the monitoring available on the
> > > client,
> > > > > > just
> > > > > > > > > doesn't seem that much better.
> > > > > > > > >
> > > > > > > > > 4. For the purgatory vs delay queue I think is arguably
> nicer
> > > to
> > > > > > reuse
> > > > > > > > the
> > > > > > > > > purgatory we just have to be ultra-conscious of
> efficiency. I
> > > > > think
> > > > > > our
> > > > > > > > > goal is to turn quotas on across the board, so at LinkedIn
> > that
> > > > > would
> > > > > > > > mean
> > > > > > > > > potentially every request will need a small delay. I
> haven't
> > > > > worked
> > > > > > out
> > > > > > > > the
> > > > > > > > > efficiency implications of this choice, so as long as we do
> > > that
> > > > > I'm
> > > > > > > > happy.
> > > > > > > > >
> > > > > > > > > -Jay
> > > > > > > > >
> > > > > > > > > On Fri, Apr 3, 2015 at 1:10 PM, Aditya Auradkar <
> > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Some responses to Jay's points.
> > > > > > > > > >
> > > > > > > > > > 1. Using commas - Cool.
> > > > > > > > > >
> > > > > > > > > > 2. Adding return flag - I'm inclined to agree with Joel
> > that
> > > > > this
> > > > > > is
> > > > > > > > good
> > > > > > > > > > to have in the initial implementation.
> > > > > > > > > >
> > > > > > > > > > 3. Config - +1. I'll remove it from the KIP. We can
> discuss
> > > this
> > > > > in
> > > > > > > > > > parallel.
> > > > > > > > > >
> > > > > > > > > > 4. Purgatory vs Delay queue - I feel that it is simpler
> to
> > > reuse
> > > > > > the
> > > > > > > > > > existing purgatories for both delayed produce and fetch
> > > > > requests.
> > > > > > > IIUC,
> > > > > > > > > all
> > > > > > > > > > we need for quotas is a minWait parameter for
> > > DelayedOperation
> > > > > (or
> > > > > > > > > > something equivalent) since there is already a max wait.
> > The
> > > > > > > completion
> > > > > > > > > > criteria can check if minWait time has elapsed before
> > > declaring
> > > > > the
> > > > > > > > > > operation complete. For this to impact performance, a
> > > > > significant
> > > > > > > > number
> > > > > > > > > of
> > > > > > > > > > clients may need to exceed their quota at the same time
> and
> > > even
> > > > > > then
> > > > > > > > I'm
> > > > > > > > > > not very clear on the scope of the impact. Two layers of
> > > delays
> > > > > > might
> > > > > > > > add
> > > > > > > > > > complexity to the implementation which I'm hoping to
> avoid.
> > > > > > > > > >
> > > > > > > > > > Aditya
> > > > > > > > > >
> > > > > > > > > > ________________________________________
> > > > > > > > > > From: Joel Koshy [jjkosh...@gmail.com]
> > > > > > > > > > Sent: Friday, April 03, 2015 12:48 PM
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > >
> > > > > > > > > > Aditya, thanks for the updated KIP and Jay/Jun thanks for
> > the
> > > > > > > > > > comments. Couple of comments in-line:
> > > > > > > > > >
> > > > > > > > > > > 2. I would advocate for adding the return flag when we
> > next
> > > > > bump
> > > > > > > the
> > > > > > > > > > > request format version just to avoid proliferation. I
> > agree
> > > > > this
> > > > > > > is a
> > > > > > > > > > good
> > > > > > > > > > > thing to know about, but at the moment I don't think we
> > > have a
> > > > > > very
> > > > > > > > > well
> > > > > > > > > > > flushed out idea of how the client would actually make
> > use
> > > of
> > > > > > this
> > > > > > > > > info.
> > > > > > > > > > I
> > > > > > > > > >
> > > > > > > > > > I'm somewhat inclined to having something appropriate off
> > the
> > > > > bat -
> > > > > > > > > > mainly because (i) clients really should know that they
> > have
> > > > > been
> > > > > > > > > > throttled (ii) a smart producer/consumer implementation
> > would
> > > > > want
> > > > > > to
> > > > > > > > > > know how much to back off. So perhaps this and
> > > config-management
> > > > > > > > > > should be moved to a separate discussion, but it would be
> > > good
> > > > > to
> > > > > > > have
> > > > > > > > > > this discussion going and incorporated into the first
> quota
> > > > > > > > > > implementation.
> > > > > > > > > >
> > > > > > > > > > > 3. Config--I think we need to generalize the topic
> stuff
> > > so we
> > > > > > can
> > > > > > > > > > override
> > > > > > > > > > > at multiple levels. We have topic and client, but I
> > suspect
> > > > > > "user"
> > > > > > > > and
> > > > > > > > > > > "broker" will also be important. I recommend we take
> > config
> > > > > stuff
> > > > > > > out
> > > > > > > > > of
> > > > > > > > > > > this KIP since we really need to fully think through a
> > > > > proposal
> > > > > > > that
> > > > > > > > > will
> > > > > > > > > > > cover all these types of overrides.
> > > > > > > > > >
> > > > > > > > > > +1 - it is definitely orthogonal to the core quota
> > > > > implementation
> > > > > > > > > > (although necessary for its operability). Having a
> > > > > config-related
> > > > > > > > > > discussion in this KIP would only draw out the discussion
> > and
> > > > > vote
> > > > > > > > > > even if the core quota design looks good to everyone.
> > > > > > > > > >
> > > > > > > > > > So basically I think we can remove the portions on
> dynamic
> > > > > config
> > > > > > as
> > > > > > > > > > well as the response format but I really think we should
> > > close
> > > > > on
> > > > > > > > > > those while the implementation is in progress and before
> > > quotas
> > > > > is
> > > > > > > > > > officially released.
> > > > > > > > > >
> > > > > > > > > > > 4. Instead of using purgatories to implement the delay
> > > would
> > > > > it
> > > > > > > make
> > > > > > > > > more
> > > > > > > > > > > sense to just use a delay queue? I think all the
> > additional
> > > > > stuff
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > purgatory other than the delay queue doesn't make sense
> > as
> > > the
> > > > > > > quota
> > > > > > > > > is a
> > > > > > > > > > > hard N ms penalty with no chance of early eviction. If
> > > there
> > > > > is
> > > > > > no
> > > > > > > > perf
> > > > > > > > > > > penalty for the full purgatory that may be fine (even
> > > good) to
> > > > > > > reuse,
> > > > > > > > > > but I
> > > > > > > > > > > haven't looked into that.
> > > > > > > > > >
> > > > > > > > > > A simple delay queue sounds good - I think Aditya was
> also
> > > > > trying
> > > > > > to
> > > > > > > > > > avoid adding a new quota purgatory. i.e., it may be
> > possible
> > > to
> > > > > use
> > > > > > > > > > the existing purgatory instances to enforce quotas. That
> > may
> > > be
> > > > > > > > > > simpler, but would be incur a slight perf penalty if too
> > many
> > > > > > clients
> > > > > > > > > > are being throttled.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Joel
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > -Jay
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Apr 3, 2015 at 10:45 AM, Aditya Auradkar <
> > > > > > > > > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Update, I added a proposal on doing dynamic client
> based
> > > > > > > > configuration
> > > > > > > > > > >> that can be used for quotas.
> > > > > > > > > > >>
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Please take a look and let me know if there are any
> > > concerns.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Aditya
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Aditya Auradkar
> > > > > > > > > > >> Sent: Friday, April 03, 2015 10:10 AM
> > > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > > >> Subject: RE: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks Jun.
> > > > > > > > > > >>
> > > > > > > > > > >> Some thoughts:
> > > > > > > > > > >>
> > > > > > > > > > >> 10) I think it is better we throttle regardless of the
> > > > > > > produce/fetch
> > > > > > > > > > >> version. This is a nice feature where clients can tell
> > if
> > > > > they
> > > > > > are
> > > > > > > > > being
> > > > > > > > > > >> throttled or not. If we only throttle newer clients,
> > then
> > > we
> > > > > > have
> > > > > > > > > > >> inconsistent behavior across clients in a multi-tenant
> > > > > cluster.
> > > > > > > > Having
> > > > > > > > > > >> quota metrics on the client side is also a nice
> > incentive
> > > to
> > > > > > > upgrade
> > > > > > > > > > client
> > > > > > > > > > >> versions.
> > > > > > > > > > >>
> > > > > > > > > > >> 11) I think we can call metric.record(fetchSize)
> before
> > > > > adding
> > > > > > the
> > > > > > > > > > >> delayedFetch request into the purgatory. This will
> give
> > us
> > > > > the
> > > > > > > > > estimated
> > > > > > > > > > >> delay of the request up-front. The timeout on the
> > > > > DelayedFetch
> > > > > > is
> > > > > > > > the
> > > > > > > > > > >> Max(maxWait, quotaDelay). The DelayedFetch completion
> > > > > criteria
> > > > > > can
> > > > > > > > > > change a
> > > > > > > > > > >> little to accomodate quotas.
> > > > > > > > > > >>
> > > > > > > > > > >> - I agree the quota code should return the estimated
> > delay
> > > > > time
> > > > > > in
> > > > > > > > > > >> QuotaViolationException.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Aditya
> > > > > > > > > > >>
> > > > > > > > > > >> ________________________________________
> > > > > > > > > > >> From: Jun Rao [j...@confluent.io]
> > > > > > > > > > >> Sent: Friday, April 03, 2015 9:16 AM
> > > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > > >> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the update.
> > > > > > > > > > >>
> > > > > > > > > > >> 10. About whether to return a new field in the
> response
> > to
> > > > > > > indicate
> > > > > > > > > > >> throttling. Earlier, the plan was to not change the
> > > response
> > > > > > > format
> > > > > > > > > and
> > > > > > > > > > >> just have a metric on the broker to indicate whether a
> > > > > clientId
> > > > > > is
> > > > > > > > > > >> throttled or not. The issue is that we don't know
> > whether
> > > a
> > > > > > > > particular
> > > > > > > > > > >> clientId instance is throttled or not (since there
> could
> > > be
> > > > > > > multiple
> > > > > > > > > > >> clients with the same clientId). Your proposal of
> adding
> > > an
> > > > > > > > > isThrottled
> > > > > > > > > > >> field in the response addresses and seems better.
> Then,
> > > do we
> > > > > > just
> > > > > > > > > > throttle
> > > > > > > > > > >> the new version of produce/fetch request or both the
> old
> > > and
> > > > > the
> > > > > > > new
> > > > > > > > > > >> versions? Also, we probably still need a separate
> metric
> > > on
> > > > > the
> > > > > > > > broker
> > > > > > > > > > side
> > > > > > > > > > >> to indicate whether a clientId is throttled or not.
> > > > > > > > > > >>
> > > > > > > > > > >> 11. Just to clarify. For fetch requests, when will
> > > > > > > > > > metric.record(fetchSize)
> > > > > > > > > > >> be called? Is it when we are ready to send the fetch
> > > response
> > > > > > > (after
> > > > > > > > > > >> minBytes and maxWait are satisfied)?
> > > > > > > > > > >>
> > > > > > > > > > >> As an implementation detail, it may be useful for the
> > > quota
> > > > > code
> > > > > > > to
> > > > > > > > > > return
> > > > > > > > > > >> an estimated delay time (to bring the measurement
> within
> > > the
> > > > > > > limit)
> > > > > > > > in
> > > > > > > > > > >> QuotaViolationException.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >>
> > > > > > > > > > >> Jun
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Apr 1, 2015 at 3:27 PM, Aditya Auradkar <
> > > > > > > > > > >> aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hey everyone,
> > > > > > > > > > >> >
> > > > > > > > > > >> > I've made changes to the KIP to capture our
> > discussions
> > > > > over
> > > > > > the
> > > > > > > > > last
> > > > > > > > > > >> > couple of weeks.
> > > > > > > > > > >> >
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >> >
> > > > > > > > > > >> > I'll start a voting thread after people have had a
> > > chance
> > > > > to
> > > > > > > > > > >> read/comment.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Aditya
> > > > > > > > > > >> >
> > > > > > > > > > >> > ________________________________________
> > > > > > > > > > >> > From: Steven Wu [stevenz...@gmail.com]
> > > > > > > > > > >> > Sent: Friday, March 20, 2015 9:14 AM
> > > > > > > > > > >> > To: dev@kafka.apache.org
> > > > > > > > > > >> > Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
> > > > > > > > > > >> >
> > > > > > > > > > >> > +1 on Jun's suggestion of maintaining one set/style
> of
> > > > > metrics
> > > > > > > at
> > > > > > > > > > broker.
> > > > > > > > > > >> > In Netflix, we have to convert the yammer metrics to
> > > servo
> > > > > > > metrics
> > > > > > > > > at
> > > > > > > > > > >> > broker. it will be painful to know some metrics are
> > in a
> > > > > > > different
> > > > > > > > > > style
> > > > > > > > > > >> > and get to be handled differently.
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Mar 20, 2015 at 8:17 AM, Jun Rao <
> > > j...@confluent.io>
> > > > >
> > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Not so sure. People who use quota will definitely
> > > want to
> > > > > > > > monitor
> > > > > > > > > > the
> > > > > > > > > > >> new
> > > > > > > > > > >> > > metrics at the client id level. Then they will
> need
> > to
> > > > > deal
> > > > > > > with
> > > > > > > > > > those
> > > > > > > > > > >> > > metrics differently from the rest of the metrics.
> It
> > > > > would
> > > > > > be
> > > > > > > > > > better if
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > can hide this complexity from the users.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Jun
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Thu, Mar 19, 2015 at 10:45 PM, Joel Koshy <
> > > > > > > > jjkosh...@gmail.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Actually thinking again - since these will be a
> > few
> > > new
> > > > > > > > metrics
> > > > > > > > > at
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > client id level (bytes in and bytes out to start
> > > with)
> > > > > > maybe
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > >> fine
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > have the two type of metrics coexist and we can
> > > migrate
> > > > > > the
> > > > > > > > > > existing
> > > > > > > > > > >> > > > metrics in parallel.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Thursday, March 19, 2015, Joel Koshy <
> > > > > > > jjkosh...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > That is a valid concern but in that case I
> think
> > > it
> > > > > > would
> > > > > > > be
> > > > > > > > > > better
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > just migrate completely to the new metrics
> > package
> > > > > > first.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Thursday, March 19, 2015, Jun Rao <
> > > > > j...@confluent.io
> > > > > > > > > > >> > > > > <javascript:_e(%7B%7D,'cvml','
> j...@confluent.io
> > > ');>>
> > > > > > > wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >> Hmm, I was thinking a bit differently on the
> > > metrics
> > > > > > > > stuff. I
> > > > > > > > > > >> think
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > >> would be confusing to have some metrics
> defined
> > > in
> > > > > the
> > > > > > > new
> > > > > > > > > > metrics
> > > > > > > > > > >> > > > package
> > > > > > > > > > >> > > > >> while some others defined in Coda Hale. Those
> > > > > metrics
> > > > > > > will
> > > > > > > > > look
> > > > > > > > > > >> > > > different
> > > > > > > > > > >> > > > >> (e.g., rates in Coda Hale will have special
> > > > > attributes
> > > > > > > such
> > > > > > > > > as
> > > > > > > > > > >> > > > >> 1-min-average). People may need different
> ways
> > to
> > > > > > export
> > > > > > > > the
> > > > > > > > > > >> metrics
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> external systems such as Graphite. So,
> instead
> > of
> > > > > using
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > >> > > metrics
> > > > > > > > > > >> > > > >> package on the broker, I was thinking that we
> > can
> > > > > just
> > > > > > > > > > implement a
> > > > > > > > > > >> > > > >> QuotaMetrics that wraps the Coda Hale
> metrics.
> > > The
> > > > > > > > > > implementation
> > > > > > > > > > >> > can
> > > > > > > > > > >> > > be
> > > > > > > > > > >> > > > >> the same as what's in the new metrics
> package.
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Thanks,
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Jun
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> On Thu, Mar 19, 2015 at 8:09 PM, Jay Kreps <
> > > > > > > > > > jay.kr...@gmail.com>
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> > Yeah I was saying was that we are blocked
> on
> > > > > picking
> > > > > > an
> > > > > > > > > > approach
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > >> > metrics but not necessarily the full
> > > conversion.
> > > > > > > Clearly
> > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > >> > pick
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> new
> > > > > > > > > > >> > > > >> > metrics package we would need to implement
> > the
> > > two
> > > > > > > > metrics
> > > > > > > > > we
> > > > > > > > > > >> want
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> quota
> > > > > > > > > > >> > > > >> > on. But the conversion of the remaining
> > metrics
> > > > > can
> > > > > > be
> > > > > > > > done
> > > > > > > > > > >> > > > >> asynchronously.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > -Jay
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > On Thu, Mar 19, 2015 at 5:56 PM, Joel
> Koshy <
> > > > > > > > > > >> jjkosh...@gmail.com>
> > > > > > > > > > >> > > > >> wrote:
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > > doesn't
> > > > > > need
> > > > > > > to
> > > > > > > > > > block
> > > > > > > > > > >> on
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Can you clarify the above? i.e., if we
> are
> > > going
> > > > > to
> > > > > > > > quota
> > > > > > > > > > on
> > > > > > > > > > >> > > > something
> > > > > > > > > > >> > > > >> > > then we would want to have migrated that
> > > metric
> > > > > > over
> > > > > > > > > > right? Or
> > > > > > > > > > >> > do
> > > > > > > > > > >> > > > you
> > > > > > > > > > >> > > > >> > > mean we don't need to complete the
> > migration
> > > of
> > > > > all
> > > > > > > > > > metrics to
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > > metrics package right?
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > I think most of us now feel that the
> delay
> > +
> > > no
> > > > > > error
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > >> good
> > > > > > > > > > >> > > > >> > > approach, but it would be good to make
> sure
> > > > > > everyone
> > > > > > > is
> > > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > >> > > same
> > > > > > > > > > >> > > > >> > > page.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > As Aditya requested a couple of days ago
> I
> > > think
> > > > > we
> > > > > > > > > should
> > > > > > > > > > go
> > > > > > > > > > >> > over
> > > > > > > > > > >> > > > >> > > this at the next KIP hangout.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Joel
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > On Thu, Mar 19, 2015 at 09:24:09AM -0700,
> > Jun
> > > > > Rao
> > > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > 1. Delay + no error seems reasonable to
> > me.
> > > > > > > However,
> > > > > > > > I
> > > > > > > > > do
> > > > > > > > > > >> feel
> > > > > > > > > > >> > > > that
> > > > > > > > > > >> > > > >> we
> > > > > > > > > > >> > > > >> > > need
> > > > > > > > > > >> > > > >> > > > to give the client an indicator that
> it's
> > > > > being
> > > > > > > > > > throttled,
> > > > > > > > > > >> > > instead
> > > > > > > > > > >> > > > >> of
> > > > > > > > > > >> > > > >> > > doing
> > > > > > > > > > >> > > > >> > > > this silently. For that, we probably
> need
> > > to
> > > > > > evolve
> > > > > > > > the
> > > > > > > > > > >> > > > >> produce/fetch
> > > > > > > > > > >> > > > >> > > > protocol to include an extra status
> field
> > > in
> > > > > the
> > > > > > > > > > response.
> > > > > > > > > > >> We
> > > > > > > > > > >> > > > >> probably
> > > > > > > > > > >> > > > >> > > need
> > > > > > > > > > >> > > > >> > > > to think more about whether we just
> want
> > to
> > > > > > return
> > > > > > > a
> > > > > > > > > > simple
> > > > > > > > > > >> > > status
> > > > > > > > > > >> > > > >> code
> > > > > > > > > > >> > > > >> > > > (e.g., 1 = throttled) or a value that
> > > > > indicates
> > > > > > how
> > > > > > > > > much
> > > > > > > > > > is
> > > > > > > > > > >> > > being
> > > > > > > > > > >> > > > >> > > throttled.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > 2. We probably need to improve the
> > > histogram
> > > > > > > support
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >> > new
> > > > > > > > > > >> > > > >> metrics
> > > > > > > > > > >> > > > >> > > > package before we can use it more
> widely
> > on
> > > > > the
> > > > > > > > server
> > > > > > > > > > side
> > > > > > > > > > >> > > (left
> > > > > > > > > > >> > > > a
> > > > > > > > > > >> > > > >> > > comment
> > > > > > > > > > >> > > > >> > > > in KAFKA-1930). I agree that this KIP
> > > doesn't
> > > > > > need
> > > > > > > to
> > > > > > > > > > block
> > > > > > > > > > >> on
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > > migration of the metrics package.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Jun
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > On Wed, Mar 18, 2015 at 4:02 PM, Aditya
> > > > > Auradkar
> > > > > > <
> > > > > > > > > > >> > > > >> > > > aaurad...@linkedin.com.invalid> wrote:
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > > Hey everyone,
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Thanks for the great discussion.
> There
> > > are
> > > > > > > > currently
> > > > > > > > > a
> > > > > > > > > > few
> > > > > > > > > > >> > > > points
> > > > > > > > > > >> > > > >> on
> > > > > > > > > > >> > > > >> > > this
> > > > > > > > > > >> > > > >> > > > > KIP that need addressing and I want
> to
> > > make
> > > > > > sure
> > > > > > > we
> > > > > > > > > > are on
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> same
> > > > > > > > > > >> > > > >> > > page
> > > > > > > > > > >> > > > >> > > > > about those.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 1. Append and delay response vs delay
> > and
> > > > > > return
> > > > > > > > > error
> > > > > > > > > > >> > > > >> > > > > - I think we've discussed the pros
> and
> > > cons
> > > > > of
> > > > > > > each
> > > > > > > > > > >> approach
> > > > > > > > > > >> > > but
> > > > > > > > > > >> > > > >> > > haven't
> > > > > > > > > > >> > > > >> > > > > chosen an approach yet. Where does
> > > everyone
> > > > > > stand
> > > > > > > > on
> > > > > > > > > > this
> > > > > > > > > > >> > > issue?
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 2. Metrics Migration and usage in
> > quotas
> > > > > > > > > > >> > > > >> > > > > - The metrics library in clients has
> a
> > > > > notion
> > > > > > of
> > > > > > > > > quotas
> > > > > > > > > > >> that
> > > > > > > > > > >> > > we
> > > > > > > > > > >> > > > >> > should
> > > > > > > > > > >> > > > >> > > > > reuse. For that to happen, we need to
> > > > > migrate
> > > > > > the
> > > > > > > > > > server
> > > > > > > > > > >> to
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> new
> > > > > > > > > > >> > > > >> > > metrics
> > > > > > > > > > >> > > > >> > > > > package.
> > > > > > > > > > >> > > > >> > > > > - Need more clarification on how to
> > > compute
> > > > > > > > > throttling
> > > > > > > > > > >> time
> > > > > > > > > > >> > > and
> > > > > > > > > > >> > > > >> > > windowing
> > > > > > > > > > >> > > > >> > > > > for quotas.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > I'm going to start a new KIP to
> discuss
> > > > > metrics
> > > > > > > > > > migration
> > > > > > > > > > >> > > > >> separately.
> > > > > > > > > > >> > > > >> > > That
> > > > > > > > > > >> > > > >> > > > > will also contain a section on
> quotas.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > 3. Dynamic Configuration management -
> > > Being
> > > > > > > > discussed
> > > > > > > > > > in
> > > > > > > > > > >> > > KIP-5.
> > > > > > > > > > >> > > > >> > > Basically
> > > > > > > > > > >> > > > >> > > > > we need something that will model
> > default
> > > > > > quotas
> > > > > > > > and
> > > > > > > > > > allow
> > > > > > > > > > >> > > > >> per-client
> > > > > > > > > > >> > > > >> > > > > overrides.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Is there something else that I'm
> > missing?
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > Aditya
> > > > > > > > > > >> > > > >> > > > >
> > ________________________________________
> > > > > > > > > > >> > > > >> > > > > From: Jay Kreps [jay.kr...@gmail.com
> ]
> > > > > > > > > > >> > > > >> > > > > Sent: Wednesday, March 18, 2015 2:10
> PM
> > > > > > > > > > >> > > > >> > > > > To: dev@kafka.apache.org
> > > > > > > > > > >> > > > >> > > > > Subject: Re: [KIP-DISCUSSION] KIP-13
> > > Quotas
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Hey Steven,
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > The current proposal is actually to
> > > enforce
> > > > > > > quotas
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> > > > > client/application level, NOT the
> topic
> > > > > level.
> > > > > > So
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > >> > have
> > > > > > > > > > >> > > a
> > > > > > > > > > >> > > > >> > service
> > > > > > > > > > >> > > > >> > > > > with a few dozen instances the quota
> is
> > > > > against
> > > > > > > all
> > > > > > > > > of
> > > > > > > > > > >> those
> > > > > > > > > > >> > > > >> > instances
> > > > > > > > > > >> > > > >> > > > > added up across all their topics. So
> > > > > actually
> > > > > > the
> > > > > > > > > > effect
> > > > > > > > > > >> > would
> > > > > > > > > > >> > > > be
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > same
> > > > > > > > > > >> > > > >> > > > > either way but throttling gives the
> > > producer
> > > > > > the
> > > > > > > > > > choice of
> > > > > > > > > > >> > > > either
> > > > > > > > > > >> > > > >> > > blocking
> > > > > > > > > > >> > > > >> > > > > or dropping.
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > -Jay
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > On Tue, Mar 17, 2015 at 10:08 AM,
> > Steven
> > > Wu
> > > > > <
> > > > > > > > > > >> > > > stevenz...@gmail.com
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > > Jay,
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > let's say an app produces to 10
> > > different
> > > > > > > topics.
> > > > > > > > > > one of
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> topic
> > > > > > > > > > >> > > > >> > is
> > > > > > > > > > >> > > > >> > > > > sent
> > > > > > > > > > >> > > > >> > > > > > from a library. due to whatever
> > > > > > condition/bug,
> > > > > > > > this
> > > > > > > > > > lib
> > > > > > > > > > >> > > starts
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > > send
> > > > > > > > > > >> > > > >> > > > > > messages over the quota. if we go
> > with
> > > the
> > > > > > > > delayed
> > > > > > > > > > >> > response
> > > > > > > > > > >> > > > >> > > approach, it
> > > > > > > > > > >> > > > >> > > > > > will cause the whole shared
> > > > > RecordAccumulator
> > > > > > > > > buffer
> > > > > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >> > > > >> filled
> > > > > > > > > > >> > > > >> > up.
> > > > > > > > > > >> > > > >> > > > > that
> > > > > > > > > > >> > > > >> > > > > > will penalize other 9 topics who
> are
> > > > > within
> > > > > > the
> > > > > > > > > > quota.
> > > > > > > > > > >> > that
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > > > > unfairness point that Ewen and I
> were
> > > > > trying
> > > > > > to
> > > > > > > > > make.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > if broker just drop the msg and
> > return
> > > an
> > > > > > > > > > error/status
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > > >> > > indicates the
> > > > > > > > > > >> > > > >> > > > > > drop and why. then producer can
> just
> > > move
> > > > > on
> > > > > > > and
> > > > > > > > > > accept
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> drop.
> > > > > > > > > > >> > > > >> > > shared
> > > > > > > > > > >> > > > >> > > > > > buffer won't be saturated and
> other 9
> > > > > topics
> > > > > > > > won't
> > > > > > > > > be
> > > > > > > > > > >> > > > penalized.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > > Steven
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > On Tue, Mar 17, 2015 at 9:44 AM,
> Jay
> > > Kreps
> > > > > <
> > > > > > > > > > >> > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > wrote:
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > > Hey Steven,
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > It is true that hitting the quota
> > > will
> > > > > > cause
> > > > > > > > > > >> > back-pressure
> > > > > > > > > > >> > > > on
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > > > > producer.
> > > > > > > > > > >> > > > >> > > > > > > But the solution is simple, a
> > > producer
> > > > > that
> > > > > > > > wants
> > > > > > > > > > to
> > > > > > > > > > >> > avoid
> > > > > > > > > > >> > > > >> this
> > > > > > > > > > >> > > > >> > > should
> > > > > > > > > > >> > > > >> > > > > > stay
> > > > > > > > > > >> > > > >> > > > > > > under its quota. In other words
> > this
> > > is
> > > > > a
> > > > > > > > > contract
> > > > > > > > > > >> > between
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> > > cluster
> > > > > > > > > > >> > > > >> > > > > > and
> > > > > > > > > > >> > > > >> > > > > > > the client, with each side having
> > > > > something
> > > > > > > to
> > > > > > > > > > uphold.
> > > > > > > > > > >> > > Quite
> > > > > > > > > > >> > > > >> > > possibly
> > > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > > >> > > > >> > > > > > > same thing will happen in the
> > > absence of
> > > > > a
> > > > > > > > > quota, a
> > > > > > > > > > >> > client
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > produces
> > > > > > > > > > >> > > > >> > > > > > an
> > > > > > > > > > >> > > > >> > > > > > > unexpected amount of load will
> hit
> > > the
> > > > > > limits
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> > > server
> > > > > > > > > > >> > > > >> and
> > > > > > > > > > >> > > > >> > > > > > experience
> > > > > > > > > > >> > > > >> > > > > > > backpressure. Quotas just allow
> you
> > > to
> > > > > set
> > > > > > > that
> > > > > > > > > > same
> > > > > > > > > > >> > limit
> > > > > > > > > > >> > > > at
> > > > > > > > > > >> > > > >> > > something
> > > > > > > > > > >> > > > >> > > > > > > lower than 100% of all resources
> on
> > > the
> > > > > > > server,
> > > > > > > > > > which
> > > > > > > > > > >> is
> > > > > > > > > > >> > > > >> useful
> > > > > > > > > > >> > > > >> > > for a
> > > > > > > > > > >> > > > >> > > > > > > shared cluster.
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > -Jay
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > On Mon, Mar 16, 2015 at 11:34 PM,
> > > Steven
> > > > > > Wu <
> > > > > > > > > > >> > > > >> > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > wait. we create one kafka
> > producer
> > > for
> > > > > > each
> > > > > > > > > > cluster.
> > > > > > > > > > >> > > each
> > > > > > > > > > >> > > > >> > > cluster can
> > > > > > > > > > >> > > > >> > > > > > > have
> > > > > > > > > > >> > > > >> > > > > > > > many topics. if producer buffer
> > got
> > > > > > filled
> > > > > > > up
> > > > > > > > > > due to
> > > > > > > > > > >> > > > delayed
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > > for
> > > > > > > > > > >> > > > >> > > > > > > > one throttled topic, won't that
> > > > > penalize
> > > > > > > > other
> > > > > > > > > > >> topics
> > > > > > > > > > >> > > > >> unfairly?
> > > > > > > > > > >> > > > >> > > it
> > > > > > > > > > >> > > > >> > > > > > seems
> > > > > > > > > > >> > > > >> > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > me that broker should just
> return
> > > > > error
> > > > > > > > without
> > > > > > > > > > >> delay.
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > sorry that I am chatting to
> > myself
> > > :)
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > On Mon, Mar 16, 2015 at 11:29
> PM,
> > > > > Steven
> > > > > > > Wu <
> > > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > > I think I can answer my own
> > > > > question.
> > > > > > > > delayed
> > > > > > > > > > >> > response
> > > > > > > > > > >> > > > >> will
> > > > > > > > > > >> > > > >> > > cause
> > > > > > > > > > >> > > > >> > > > > the
> > > > > > > > > > >> > > > >> > > > > > > > > producer buffer to be full,
> > which
> > > > > then
> > > > > > > > result
> > > > > > > > > > in
> > > > > > > > > > >> > > either
> > > > > > > > > > >> > > > >> > thread
> > > > > > > > > > >> > > > >> > > > > > blocking
> > > > > > > > > > >> > > > >> > > > > > > > or
> > > > > > > > > > >> > > > >> > > > > > > > > message drop.
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > > On Mon, Mar 16, 2015 at 11:24
> > PM,
> > > > > > Steven
> > > > > > > > Wu <
> > > > > > > > > > >> > > > >> > > stevenz...@gmail.com>
> > > > > > > > > > >> > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >> please correct me if I am
> > > missing
> > > > > sth
> > > > > > > > here.
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > >> > > understanding
> > > > > > > > > > >> > > > >> > > > > > how
> > > > > > > > > > >> > > > >> > > > > > > > >> would throttle work without
> > > > > > > > > > cooperation/back-off
> > > > > > > > > > >> > from
> > > > > > > > > > >> > > > >> > > producer.
> > > > > > > > > > >> > > > >> > > > > new
> > > > > > > > > > >> > > > >> > > > > > > Java
> > > > > > > > > > >> > > > >> > > > > > > > >> producer supports
> non-blocking
> > > API.
> > > > > > why
> > > > > > > > > would
> > > > > > > > > > >> > delayed
> > > > > > > > > > >> > > > >> > > response be
> > > > > > > > > > >> > > > >> > > > > > able
> > > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > >> slow down producer? producer
> > > will
> > > > > > > continue
> > > > > > > > > to
> > > > > > > > > > >> fire
> > > > > > > > > > >> > > > async
> > > > > > > > > > >> > > > >> > > sends.
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >> On Mon, Mar 16, 2015 at
> 10:58
> > > PM,
> > > > > > > Guozhang
> > > > > > > > > > Wang <
> > > > > > > > > > >> > > > >> > > > > wangg...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> I think we are really
> > > discussing
> > > > > two
> > > > > > > > > separate
> > > > > > > > > > >> > issues
> > > > > > > > > > >> > > > >> here:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> 1. Whether we should a)
> > > > > > > > > > >> > > > >> > > > >
> > > append-then-block-then-returnOKButThrottled
> > > > > > > > > > >> > > > >> > > > > > > or
> > > > > > > > > > >> > > > >> > > > > > > > b)
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > block-then-returnFailDuetoThrottled
> > > > > > for
> > > > > > > > > quota
> > > > > > > > > > >> > > actions
> > > > > > > > > > >> > > > on
> > > > > > > > > > >> > > > >> > > produce
> > > > > > > > > > >> > > > >> > > > > > > > >>> requests.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Both these approaches
> assume
> > > some
> > > > > > kind
> > > > > > > of
> > > > > > > > > > >> > > > >> well-behaveness
> > > > > > > > > > >> > > > >> > of
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > > > > clients:
> > > > > > > > > > >> > > > >> > > > > > > > >>> option a) assumes the
> client
> > > sets
> > > > > an
> > > > > > > > proper
> > > > > > > > > > >> > timeout
> > > > > > > > > > >> > > > >> value
> > > > > > > > > > >> > > > >> > > while
> > > > > > > > > > >> > > > >> > > > > can
> > > > > > > > > > >> > > > >> > > > > > > > just
> > > > > > > > > > >> > > > >> > > > > > > > >>> ignore "OKButThrottled"
> > > response,
> > > > > > while
> > > > > > > > > > option
> > > > > > > > > > >> b)
> > > > > > > > > > >> > > > >> assumes
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > client
> > > > > > > > > > >> > > > >> > > > > > > > >>> handles the
> > > "FailDuetoThrottled"
> > > > > > > > > > appropriately.
> > > > > > > > > > >> > For
> > > > > > > > > > >> > > > any
> > > > > > > > > > >> > > > >> > > malicious
> > > > > > > > > > >> > > > >> > > > > > > > clients
> > > > > > > > > > >> > > > >> > > > > > > > >>> that, for example, just
> keep
> > > > > retrying
> > > > > > > > > either
> > > > > > > > > > >> > > > >> intentionally
> > > > > > > > > > >> > > > >> > or
> > > > > > > > > > >> > > > >> > > > > not,
> > > > > > > > > > >> > > > >> > > > > > > > >>> neither
> > > > > > > > > > >> > > > >> > > > > > > > >>> of these approaches are
> > > actually
> > > > > > > > effective.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> 2. For "OKButThrottled" and
> > > > > > > > > > "FailDuetoThrottled"
> > > > > > > > > > >> > > > >> responses,
> > > > > > > > > > >> > > > >> > > shall
> > > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> encode
> > > > > > > > > > >> > > > >> > > > > > > > >>> them as error codes or
> > augment
> > > the
> > > > > > > > protocol
> > > > > > > > > > to
> > > > > > > > > > >> > use a
> > > > > > > > > > >> > > > >> > separate
> > > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > > >> > > > >> > > > > > > > >>> indicating "status codes".
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Today we have already
> > > incorporated
> > > > > > some
> > > > > > > > > > status
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > as
> > > > > > > > > > >> > > > >> > error
> > > > > > > > > > >> > > > >> > > > > codes
> > > > > > > > > > >> > > > >> > > > > > in
> > > > > > > > > > >> > > > >> > > > > > > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> responses, e.g.
> > > > > ReplicaNotAvailable
> > > > > > in
> > > > > > > > > > >> > > > MetadataResponse,
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > pros
> > > > > > > > > > >> > > > >> > > > > > of
> > > > > > > > > > >> > > > >> > > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> is of course using a single
> > > field
> > > > > for
> > > > > > > > > > response
> > > > > > > > > > >> > > status
> > > > > > > > > > >> > > > >> like
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > HTTP
> > > > > > > > > > >> > > > >> > > > > > > > >>> status
> > > > > > > > > > >> > > > >> > > > > > > > >>> codes, while the cons is
> that
> > > it
> > > > > > > requires
> > > > > > > > > > >> clients
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > >> handle
> > > > > > > > > > >> > > > >> > > the
> > > > > > > > > > >> > > > >> > > > > > error
> > > > > > > > > > >> > > > >> > > > > > > > >>> codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> carefully.
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> I think maybe we can
> actually
> > > > > extend
> > > > > > > the
> > > > > > > > > > >> > single-code
> > > > > > > > > > >> > > > >> > > approach to
> > > > > > > > > > >> > > > >> > > > > > > > overcome
> > > > > > > > > > >> > > > >> > > > > > > > >>> its drawbacks, that is,
> wrap
> > > the
> > > > > > error
> > > > > > > > > codes
> > > > > > > > > > >> > > semantics
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > users
> > > > > > > > > > >> > > > >> > > > > > > so
> > > > > > > > > > >> > > > >> > > > > > > > >>> that
> > > > > > > > > > >> > > > >> > > > > > > > >>> users do not need to handle
> > the
> > > > > codes
> > > > > > > > > > >> one-by-one.
> > > > > > > > > > >> > > More
> > > > > > > > > > >> > > > >> > > > > concretely,
> > > > > > > > > > >> > > > >> > > > > > > > >>> following Jay's example the
> > > client
> > > > > > > could
> > > > > > > > > > write
> > > > > > > > > > >> > sth.
> > > > > > > > > > >> > > > like
> > > > > > > > > > >> > > > >> > > this:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or
> the
> > > > > > code
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> > > simply
> > > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // throttled, transient
> > error,
> > > > > > > etc:
> > > > > > > > > > retry
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors,
> etc:
> > > > > > > > notify /
> > > > > > > > > > >> > > terminate
> > > > > > > > > > >> > > > /
> > > > > > > > > > >> > > > >> > other
> > > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Only when the clients
> really
> > > want
> > > > > to
> > > > > > > > > handle,
> > > > > > > > > > for
> > > > > > > > > > >> > > > example
> > > > > > > > > > >> > > > >> > > > > > > > >>> FailDuetoThrottled
> > > > > > > > > > >> > > > >> > > > > > > > >>> status code specifically,
> it
> > > needs
> > > > > > to:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> if(error.isOK())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // status code is good or
> the
> > > > > > code
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> > > simply
> > > > > > > > > > >> > > > >> > > ignored for
> > > > > > > > > > >> > > > >> > > > > > > this
> > > > > > > > > > >> > > > >> > > > > > > > >>> request type, process the
> > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error ==
> > > > > > FailDuetoThrottled )
> > > > > > > > > > >> > > > >> > > > > > > > >>> // throttled: log it
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.needsRetry())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // transient error, etc:
> > retry
> > > > > > > > > > >> > > > >> > > > > > > > >>> else if(error.isFatal())
> > > > > > > > > > >> > > > >> > > > > > > > >>> // non-retriable errors,
> etc:
> > > > > > > > notify /
> > > > > > > > > > >> > > terminate
> > > > > > > > > > >> > > > /
> > > > > > > > > > >> > > > >> > other
> > > > > > > > > > >> > > > >> > > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> -----------------
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> And for implementation we
> can
> > > > > > probably
> > > > > > > > > group
> > > > > > > > > > the
> > > > > > > > > > >> > > codes
> > > > > > > > > > >> > > > >> > > > > accordingly
> > > > > > > > > > >> > > > >> > > > > > > like
> > > > > > > > > > >> > > > >> > > > > > > > >>> HTTP status code such that
> we
> > > can
> > > > > do:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> boolean Error.isOK() {
> > > > > > > > > > >> > > > >> > > > > > > > >>> return code < 300 && code
> >=
> > > 200;
> > > > > > > > > > >> > > > >> > > > > > > > >>> }
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> Guozhang
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> On Mon, Mar 16, 2015 at
> 10:24
> > > PM,
> > > > > > Ewen
> > > > > > > > > > >> > > > Cheslack-Postava
> > > > > > > > > > >> > > > >> <
> > > > > > > > > > >> > > > >> > > > > > > > >>> e...@confluent.io>
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Agreed that trying to
> > > shoehorn
> > > > > > > > non-error
> > > > > > > > > > codes
> > > > > > > > > > >> > > into
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > field
> > > > > > > > > > >> > > > >> > > > > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > bad idea. It makes it
> *way*
> > > too
> > > > > > easy
> > > > > > > to
> > > > > > > > > > write
> > > > > > > > > > >> > code
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > looks
> > > > > > > > > > >> > > > >> > > > > > (and
> > > > > > > > > > >> > > > >> > > > > > > > >>> should
> > > > > > > > > > >> > > > >> > > > > > > > >>> > be) correct but is
> actually
> > > > > > > incorrect.
> > > > > > > > If
> > > > > > > > > > >> > > > necessary, I
> > > > > > > > > > >> > > > >> > > think
> > > > > > > > > > >> > > > >> > > > > it's
> > > > > > > > > > >> > > > >> > > > > > > > much
> > > > > > > > > > >> > > > >> > > > > > > > >>> > better to to spend a
> couple
> > > of
> > > > > > extra
> > > > > > > > > bytes
> > > > > > > > > > to
> > > > > > > > > > >> > > encode
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > > > information
> > > > > > > > > > >> > > > >> > > > > > > > >>> > separately (a "status" or
> > > > > "warning"
> > > > > > > > > > section of
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > > response).
> > > > > > > > > > >> > > > >> > > > > An
> > > > > > > > > > >> > > > >> > > > > > > > >>> indication
> > > > > > > > > > >> > > > >> > > > > > > > >>> > that throttling is
> > occurring
> > > is
> > > > > > > > something
> > > > > > > > > > I'd
> > > > > > > > > > >> > > expect
> > > > > > > > > > >> > > > >> to
> > > > > > > > > > >> > > > >> > be
> > > > > > > > > > >> > > > >> > > > > > > indicated
> > > > > > > > > > >> > > > >> > > > > > > > >>> by a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > bit flag in the response
> > > rather
> > > > > > than
> > > > > > > as
> > > > > > > > > an
> > > > > > > > > > >> error
> > > > > > > > > > >> > > > code.
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Gwen - I think an error
> > code
> > > > > makes
> > > > > > > > sense
> > > > > > > > > > when
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> request
> > > > > > > > > > >> > > > >> > > > > > actually
> > > > > > > > > > >> > > > >> > > > > > > > >>> failed.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Option B, which Jun was
> > > > > advocating,
> > > > > > > > would
> > > > > > > > > > have
> > > > > > > > > > >> > > > >> appended
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > > messages
> > > > > > > > > > >> > > > >> > > > > > > > >>> > successfully. If the
> > > > > rate-limiting
> > > > > > > case
> > > > > > > > > > you're
> > > > > > > > > > >> > > > talking
> > > > > > > > > > >> > > > >> > > about
> > > > > > > > > > >> > > > >> > > > > had
> > > > > > > > > > >> > > > >> > > > > > > > >>> > successfully committed
> the
> > > > > > messages,
> > > > > > > I
> > > > > > > > > > would
> > > > > > > > > > >> say
> > > > > > > > > > >> > > > >> that's
> > > > > > > > > > >> > > > >> > > also a
> > > > > > > > > > >> > > > >> > > > > > bad
> > > > > > > > > > >> > > > >> > > > > > > > use
> > > > > > > > > > >> > > > >> > > > > > > > >>> of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > error codes.
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > On Mon, Mar 16, 2015 at
> > 10:16
> > > > > PM,
> > > > > > > Gwen
> > > > > > > > > > >> Shapira <
> > > > > > > > > > >> > > > >> > > > > > > > gshap...@cloudera.com>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > We discussed an error
> > code
> > > for
> > > > > > > > > > rate-limiting
> > > > > > > > > > >> > > > (which
> > > > > > > > > > >> > > > >> I
> > > > > > > > > > >> > > > >> > > think
> > > > > > > > > > >> > > > >> > > > > > made
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > sense), isn't it a
> > similar
> > > > > case?
> > > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > On Mon, Mar 16, 2015 at
> > > 10:10
> > > > > PM,
> > > > > > > Jay
> > > > > > > > > > Kreps
> > > > > > > > > > >> <
> > > > > > > > > > >> > > > >> > > > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > My concern is that as
> > > soon
> > > > > as
> > > > > > you
> > > > > > > > > start
> > > > > > > > > > >> > > encoding
> > > > > > > > > > >> > > > >> > > non-error
> > > > > > > > > > >> > > > >> > > > > > > > response
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > information into
> error
> > > codes
> > > > > > the
> > > > > > > > next
> > > > > > > > > > >> > question
> > > > > > > > > > >> > > > is
> > > > > > > > > > >> > > > >> > what
> > > > > > > > > > >> > > > >> > > to
> > > > > > > > > > >> > > > >> > > > > do
> > > > > > > > > > >> > > > >> > > > > > if
> > > > > > > > > > >> > > > >> > > > > > > > two
> > > > > > > > > > >> > > > >> > > > > > > > >>> > such
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > codes apply (i.e. you
> > > have a
> > > > > > > > replica
> > > > > > > > > > down
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > quota'd). I
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > think I am trying to
> > > argue
> > > > > that
> > > > > > > > error
> > > > > > > > > > >> should
> > > > > > > > > > >> > > > mean
> > > > > > > > > > >> > > > >> > "why
> > > > > > > > > > >> > > > >> > > we
> > > > > > > > > > >> > > > >> > > > > > > failed
> > > > > > > > > > >> > > > >> > > > > > > > >>> your
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > request", for which
> > there
> > > > > will
> > > > > > > > really
> > > > > > > > > > only
> > > > > > > > > > >> > be
> > > > > > > > > > >> > > > one
> > > > > > > > > > >> > > > >> > > reason,
> > > > > > > > > > >> > > > >> > > > > and
> > > > > > > > > > >> > > > >> > > > > > > any
> > > > > > > > > > >> > > > >> > > > > > > > >>> other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > useful information we
> > > want
> > > > > to
> > > > > > > send
> > > > > > > > > > back is
> > > > > > > > > > >> > > just
> > > > > > > > > > >> > > > >> > another
> > > > > > > > > > >> > > > >> > > > > field
> > > > > > > > > > >> > > > >> > > > > > > in
> > > > > > > > > > >> > > > >> > > > > > > > >>> the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > response.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > -Jay
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > > On Mon, Mar 16, 2015
> at
> > > 9:51
> > > > > > PM,
> > > > > > > > Gwen
> > > > > > > > > > >> > Shapira
> > > > > > > > > > >> > > <
> > > > > > > > > > >> > > > >> > > > > > > > >>> gshap...@cloudera.com>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> I think its not too
> > > late to
> > > > > > > > reserve
> > > > > > > > > a
> > > > > > > > > > set
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > >> error
> > > > > > > > > > >> > > > >> > > codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> (200-299?)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> for "non-error"
> codes.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It won't be backward
> > > > > > compatible
> > > > > > > > > (i.e.
> > > > > > > > > > >> > clients
> > > > > > > > > > >> > > > >> that
> > > > > > > > > > >> > > > >> > > > > currently
> > > > > > > > > > >> > > > >> > > > > > > do
> > > > > > > > > > >> > > > >> > > > > > > > >>> "else
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> throw" will throw on
> > > > > > > non-errors),
> > > > > > > > > but
> > > > > > > > > > >> > perhaps
> > > > > > > > > > >> > > > its
> > > > > > > > > > >> > > > >> > > > > > worthwhile.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> On Mon, Mar 16, 2015
> > at
> > > > > 9:42
> > > > > > PM,
> > > > > > > > Jay
> > > > > > > > > > >> Kreps
> > > > > > > > > > >> > <
> > > > > > > > > > >> > > > >> > > > > > > jay.kr...@gmail.com
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Hey Jun,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I'd really really
> > > really
> > > > > > like
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > >> > > that.
> > > > > > > > > > >> > > > >> > Having
> > > > > > > > > > >> > > > >> > > just
> > > > > > > > > > >> > > > >> > > > > > > > spent a
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > bunch of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > time on the
> clients,
> > > > > using
> > > > > > the
> > > > > > > > > error
> > > > > > > > > > >> > codes
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> > encode
> > > > > > > > > > >> > > > >> > > > > other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > information
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > about the response
> > is
> > > > > super
> > > > > > > > > > dangerous.
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > >> error
> > > > > > > > > > >> > > > >> > > > > handling
> > > > > > > > > > >> > > > >> > > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> one of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > hardest parts of
> the
> > > > > client
> > > > > > > > > > (Guozhang
> > > > > > > > > > >> > chime
> > > > > > > > > > >> > > > in
> > > > > > > > > > >> > > > >> > > here).
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Generally the
> error
> > > > > handling
> > > > > > > > looks
> > > > > > > > > > like
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > if(error == none)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // good, process
> the
> > > > > > > > request
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > > KNOWN_ERROR_1)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> > error
> > > 1
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else if(error ==
> > > > > > > > KNOWN_ERROR_2)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > // handle known
> > error
> > > 2
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > else
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > throw
> > > > > > > > > > >> > > Errors.forCode(error).exception();
> > > > > > > > > > >> > > > >> //
> > > > > > > > > > >> > > > >> > or
> > > > > > > > > > >> > > > >> > > some
> > > > > > > > > > >> > > > >> > > > > > > other
> > > > > > > > > > >> > > > >> > > > > > > > >>> > default
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > behavior
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > This works because
> > we
> > > > > have a
> > > > > > > > > > convention
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > >> and
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> something
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > that
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > prevented your
> > getting
> > > > > the
> > > > > > > > > response
> > > > > > > > > > so
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> default
> > > > > > > > > > >> > > > >> > > > > > handling
> > > > > > > > > > >> > > > >> > > > > > > > >>> case is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > sane
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > and forward
> > > compatible.
> > > > > It
> > > > > > is
> > > > > > > > > > tempting
> > > > > > > > > > >> to
> > > > > > > > > > >> > > use
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > code
> > > > > > > > > > >> > > > >> > > > > > > > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > convey
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > information in the
> > > > > success
> > > > > > > case.
> > > > > > > > > For
> > > > > > > > > > >> > > example
> > > > > > > > > > >> > > > we
> > > > > > > > > > >> > > > >> > > could
> > > > > > > > > > >> > > > >> > > > > use
> > > > > > > > > > >> > > > >> > > > > > > > error
> > > > > > > > > > >> > > > >> > > > > > > > >>> > codes
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > encode whether
> > quotas
> > > > > were
> > > > > > > > > enforced,
> > > > > > > > > > >> > > whether
> > > > > > > > > > >> > > > >> the
> > > > > > > > > > >> > > > >> > > request
> > > > > > > > > > >> > > > >> > > > > > was
> > > > > > > > > > >> > > > >> > > > > > > > >>> served
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > out
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> of
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > cache, whether the
> > > stock
> > > > > > > market
> > > > > > > > is
> > > > > > > > > > up
> > > > > > > > > > >> > > today,
> > > > > > > > > > >> > > > or
> > > > > > > > > > >> > > > >> > > > > whatever.
> > > > > > > > > > >> > > > >> > > > > > > The
> > > > > > > > > > >> > > > >> > > > > > > > >>> > problem
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > that since these
> are
> > > not
> > > > > > > errors
> > > > > > > > as
> > > > > > > > > > far
> > > > > > > > > > >> as
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > client is
> > > > > > > > > > >> > > > >> > > > > > > > >>> concerned it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> should
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > not throw an
> > exception
> > > > > but
> > > > > > > > process
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> response,
> > > > > > > > > > >> > > > >> > > but now
> > > > > > > > > > >> > > > >> > > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> created
> > > > > > > > > > >> > > > >> > > > > > > > >>> > an
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > explicit
> requirement
> > > that
> > > > > > that
> > > > > > > > > > error be
> > > > > > > > > > >> > > > handled
> > > > > > > > > > >> > > > >> > > > > explicitly
> > > > > > > > > > >> > > > >> > > > > > > > >>> since it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > is
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > different. I
> really
> > > think
> > > > > > that
> > > > > > > > > this
> > > > > > > > > > >> kind
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > >> > > information
> > > > > > > > > > >> > > > >> > > > > is
> > > > > > > > > > >> > > > >> > > > > > > not
> > > > > > > > > > >> > > > >> > > > > > > > >>> an
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > error,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> it
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is just
> information,
> > > and
> > > > > if
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > it
> > > > > > > > > > >> in
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > >> > > response
> > > > > > > > > > >> > > > >> > > > > we
> > > > > > > > > > >> > > > >> > > > > > > > >>> should do
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > right thing and
> add
> > a
> > > new
> > > > > > > field
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> response.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > I think you saw
> the
> > > Samza
> > > > > > bug
> > > > > > > > that
> > > > > > > > > > was
> > > > > > > > > > >> > > > >> literally
> > > > > > > > > > >> > > > >> > an
> > > > > > > > > > >> > > > >> > > > > > example
> > > > > > > > > > >> > > > >> > > > > > > of
> > > > > > > > > > >> > > > >> > > > > > > > >>> this
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > happening and
> > leading
> > > to
> > > > > an
> > > > > > > > > infinite
> > > > > > > > > > >> > retry
> > > > > > > > > > >> > > > >> loop.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > Further more I
> > really
> > > > > want
> > > > > > to
> > > > > > > > > > emphasize
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > >> > hitting
> > > > > > > > > > >> > > > >> > > > > your
> > > > > > > > > > >> > > > >> > > > > > > > quota
> > > > > > > > > > >> > > > >> > > > > > > > >>> in
> > > > > > > > > > >> > > > >> > > > > > > > >>> > the
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > design that Adi
> has
> > > > > proposed
> > > > > > > is
> > > > > > > > > > >> actually
> > > > > > > > > > >> > > not
> > > > > > > > > > >> > > > an
> > > > > > > > > > >> > > > >> > > error
> > > > > > > > > > >> > > > >> > > > > > > > condition
> > > > > > > > > > >> > > > >> > > > > > > > >>> at
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > all.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> It
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > is totally
> > reasonable
> > > in
> > > > > any
> > > > > > > > > > bootstrap
> > > > > > > > > > >> > > > >> situation
> > > > > > > > > > >> > > > >> > to
> > > > > > > > > > >> > > > >> > > > > > > > >>> intentionally
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > want to
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > run at the limit
> the
> > > > > system
> > > > > > > > > imposes
> > > > > > > > > > on
> > > > > > > > > > >> > you.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > -Jay
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> > On Mon, Mar 16,
> 2015
> > > at
> > > > > 4:27
> > > > > > > PM,
> > > > > > > > > Jun
> > > > > > > > > > >> Rao
> > > > > > > > > > >> > <
> > > > > > > > > > >> > > > >> > > > > > j...@confluent.io>
> > > > > > > > > > >> > > > >> > > > > > > > >>> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> It's probably
> > useful
> > > for
> > > > > a
> > > > > > > > client
> > > > > > > > > > to
> > > > > > > > > > >> > know
> > > > > > > > > > >> > > > >> whether
> > > > > > > > > > >> > > > >> > > its
> > > > > > > > > > >> > > > >> > > > > > > > requests
> > > > > > > > > > >> > > > >> > > > > > > > >>> are
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> throttled or not
> > > (e.g.,
> > > > > for
> > > > > > > > > > monitoring
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > >> > > alerting).
> > > > > > > > > > >> > > > >> > > > > > From
> > > > > > > > > > >> > > > >> > > > > > > > that
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> perspective,
> > option B
> > > > > > (delay
> > > > > > > > the
> > > > > > > > > > >> > requests
> > > > > > > > > > >> > > > and
> > > > > > > > > > >> > > > >> > > return an
> > > > > > > > > > >> > > > >> > > > > > > > error)
> > > > > > > > > > >> > > > >> > > > > > > > >>> > seems
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> better.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Thanks,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> Jun
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> On Wed, Mar 4,
> 2015
> > > at
> > > > > 3:51
> > > > > > > PM,
> > > > > > > > > > Aditya
> > > > > > > > > > >> > > > >> Auradkar <
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > aaurad...@linkedin.com.invalid
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Posted a KIP
> for
> > > > > quotas
> > > > > > in
> > > > > > > > > kafka.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >>
> > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Appreciate any
> > > > > feedback.
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> > Aditya
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >> >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>> > >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>> > --
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Thanks,
> > > > > > > > > > >> > > > >> > > > > > > > >>> > Ewen
> > > > > > > > > > >> > > > >> > > > > > > > >>> >
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>> --
> > > > > > > > > > >> > > > >> > > > > > > > >>> -- Guozhang
> > > > > > > > > > >> > > > >> > > > > > > > >>>
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >>
> > > > > > > > > > >> > > > >> > > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > > >
> > > > > > > > > > >> > > > >> > > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > --
> > > > > > > > > > >> > > > > Sent from Gmail Mobile
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > --
> > > > > > > > > > >> > > > Sent from Gmail Mobile
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>



--
-- Guozhang