Repository: kafka Updated Branches: refs/heads/trunk d1d71aa29 -> 670193f22
KAFKA-4957; Request rate quotas documentation Author: Rajini Sivaram <[email protected]> Reviewers: Jun Rao <[email protected]> Closes #3069 from rajinisivaram/KAFKA-4957 Project: http://git-wip-us.apache.org/repos/asf/kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/670193f2 Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/670193f2 Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/670193f2 Branch: refs/heads/trunk Commit: 670193f22d9e05a130024c1bcc7408ac2f736fba Parents: d1d71aa Author: Rajini Sivaram <[email protected]> Authored: Tue May 16 16:36:25 2017 -0700 Committer: Jun Rao <[email protected]> Committed: Tue May 16 16:36:25 2017 -0700 ---------------------------------------------------------------------- docs/design.html | 35 +++++++++++++++++++++++++++++------ docs/ops.html | 41 +++++++++++++++++++++++++++-------------- 2 files changed, 56 insertions(+), 20 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kafka/blob/670193f2/docs/design.html ---------------------------------------------------------------------- diff --git a/docs/design.html b/docs/design.html index aca0d26..c061cf4 100644 --- a/docs/design.html +++ b/docs/design.html @@ -524,12 +524,18 @@ <h3><a id="design_quotas" href="#design_quotas">4.9 Quotas</a></h3> <p> - Starting in 0.9, the Kafka cluster has the ability to enforce quotas on produce and fetch requests. Quotas are basically byte-rate thresholds defined per group of clients sharing a quota. + Kafka cluster has the ability to enforce quotas on requests to control the broker resources used by clients. Two types + of client quotas can be enforced by Kafka brokers for each group of clients sharing a quota: + <ol> + <li>Network bandwidth quotas define byte-rate thresholds (since 0.9)</li> + <li>Request rate quotas define CPU utilization thresholds as a percentage of network and I/O threads (since 0.11)</li> + </ol> </p> <h4><a id="design_quotasnecessary" href="#design_quotasnecessary">Why are quotas necessary?</a></h4> <p> - It is possible for producers and consumers to produce/consume very high volumes of data and thus monopolize broker resources, cause network saturation and generally DOS other clients and the brokers themselves. + It is possible for producers and consumers to produce/consume very high volumes of data or generate requests at a very high + rate and thus monopolize broker resources, cause network saturation and generally DOS other clients and the brokers themselves. Having quotas protects against these issues and is all the more important in large multi-tenant clusters where a small set of badly behaved clients can degrade user experience for the well behaved ones. In fact, when running Kafka as a service this even makes it possible to enforce API limits according to an agreed upon contract. </p> @@ -563,13 +569,30 @@ <li>/config/clients/<default></li> </ol> - Broker properties (quota.producer.default, quota.consumer.default) can also be used to set defaults for client-id groups. These properties are being deprecated and will be removed in a later release. + Broker properties (quota.producer.default, quota.consumer.default) can also be used to set defaults of network bandwidth quotas for client-id groups. These properties are being deprecated and will be removed in a later release. Default quotas for client-id can be set in Zookeeper similar to the other quota overrides and defaults. </p> - <h4><a id="design_quotasenforcement" href="#design_quotasenforcement">Enforcement</a></h4> + <h4><a id="design_quotasbandwidth" href="#design_quotasbandwidth">Network Bandwidth Quotas</a></h4> <p> + Network bandwidth quotas are defined as the byte rate threshold for each group of clients sharing a quota. By default, each unique client group receives a fixed quota in bytes/sec as configured by the cluster. - This quota is defined on a per-broker basis. Each client can publish/fetch a maximum of X bytes/sec per broker before it gets throttled. We decided that defining these quotas per broker is much better than + This quota is defined on a per-broker basis. Each group of clients can publish/fetch a maximum of X bytes/sec + per broker before clients are throttled. + </p> + <h4><a id="design_quotascpu" href="#design_quotascpu">Request Rate Quotas</a></h4> + <p> + Request rate quotas are defined as the percentage of time a client can utilize on request handler I/O + threads and network threads of each broker within a quota window. A quota of <tt>n%</tt> represents + <tt>n%</tt> of one thread, so the quota is out of a total capacity of <tt>((num.io.threads + num.network.threads) * 100)%</tt>. + Each group of clients may use a total percentage of upto <tt>n%</tt> across all I/O and network threads in a quota + window before being throttled. Since the number of threads allocated for I/O and network threads are typically based + on the number of cores available on the broker host, request rate quotas represent the total percentage of CPU + that may be used by each group of clients sharing the quota. + </p> + <h4><a id="design_quotasenforcement" href="#design_quotasenforcement">Enforcement</a></h4> + <p> + By default, each unique client group receives a fixed quota as configured by the cluster. + This quota is defined on a per-broker basis. Each client can utilize this quota per broker before it gets throttled. We decided that defining these quotas per broker is much better than having a fixed cluster wide bandwidth per client because that would require a mechanism to share client quota usage among all the brokers. This can be harder to get right than the quota implementation itself! </p> <p> @@ -579,7 +602,7 @@ can exacerbate the very problem quotas are trying to solve. </p> <p> - Client byte rate is measured over multiple small windows (e.g. 30 windows of 1 second each) in order to detect and correct quota violations quickly. Typically, having large measurement windows + Byte-rate and thread utilization are measured over multiple small windows (e.g. 30 windows of 1 second each) in order to detect and correct quota violations quickly. Typically, having large measurement windows (for e.g. 10 windows of 30 seconds each) leads to large bursts of traffic followed by long delays which is not great in terms of user experience. </p> </script> http://git-wip-us.apache.org/repos/asf/kafka/blob/670193f2/docs/ops.html ---------------------------------------------------------------------- diff --git a/docs/ops.html b/docs/ops.html index 7be9939..eface56 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -449,19 +449,19 @@ <p> Configure custom quota for (user=user1, client-id=clientA): <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type users --entity-name user1 --entity-type clients --entity-name clientA + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-name clientA Updated config for entity: user-principal 'user1', client-id 'clientA'. </pre> Configure custom quota for user=user1: <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type users --entity-name user1 + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 Updated config for entity: user-principal 'user1'. </pre> Configure custom quota for client-id=clientA: <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type clients --entity-name clientA + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-name clientA Updated config for entity: client-id 'clientA'. </pre> @@ -469,48 +469,48 @@ <p> Configure default client-id quota for user=userA: <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type users --entity-name user1 --entity-type clients --entity-default + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-name user1 --entity-type clients --entity-default Updated config for entity: user-principal 'user1', default client-id. </pre> Configure default quota for user: <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type users --entity-default + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type users --entity-default Updated config for entity: default user-principal. </pre> Configure default quota for client-id: <pre> - > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-type clients --entity-default + > bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' --entity-type clients --entity-default Updated config for entity: default client-id. </pre> Here's how to describe the quota for a given (user, client-id): <pre> > bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users --entity-name user1 --entity-type clients --entity-name clientA - Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048 + Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 </pre> Describe quota for a given user: <pre> > bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users --entity-name user1 - Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048 + Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 </pre> Describe quota for a given client-id: <pre> > bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type clients --entity-name clientA - Configs for client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048 + Configs for client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 </pre> If entity name is not specified, all entities of the specified type are described. For example, describe all users: <pre> > bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users - Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048 - Configs for default user-principal are producer_byte_rate=1024,consumer_byte_rate=2048 + Configs for user-principal 'user1' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + Configs for default user-principal are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 </pre> Similarly for (user, client): <pre> > bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-type users --entity-type clients - Configs for user-principal 'user1', default client-id are producer_byte_rate=1024,consumer_byte_rate=2048 - Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048 + Configs for user-principal 'user1', default client-id are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 + Configs for user-principal 'user1', client-id 'clientA' are producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200 </pre> <p> It is possible to set default quotas that apply to all client-ids by setting these configs on the brokers. These properties are applied only if quota overrides or defaults are not configured in Zookeeper. By default, each client-id receives an unlimited quota. The following sets the default quota per producer and consumer client-id to 10MB/sec. @@ -834,12 +834,25 @@ <td>between 0 and 1, ideally > 0.3</td> </tr> <tr> - <td>Quota metrics per (user, client-id), user or client-id</td> + <td>Bandwidth quota metrics per (user, client-id), user or client-id</td> <td>kafka.server:type={Produce|Fetch},user=([-.\w]+),client-id=([-.\w]+)</td> <td>Two attributes. throttle-time indicates the amount of time in ms the client was throttled. Ideally = 0. byte-rate indicates the data produce/consume rate of the client in bytes/sec. For (user, client-id) quotas, both user and client-id are specified. If per-client-id quota is applied to the client, user is not specified. If per-user quota is applied, client-id is not specified.</td> </tr> + <tr> + <td>Request quota metrics per (user, client-id), user or client-id</td> + <td>kafka.server:type=Request,user=([-.\w]+),client-id=([-.\w]+)</td> + <td>Two attributes. throttle-time indicates the amount of time in ms the client was throttled. Ideally = 0. + request-time indicates the percentage of time spent in broker network and I/O threads to process requests from client group. + For (user, client-id) quotas, both user and client-id are specified. If per-client-id quota is applied to the client, user is not specified. If per-user quota is applied, client-id is not specified.</td> + </tr> + <tr> + <td>Requests exempt from throttling</td> + <td>kafka.server:type=Request</td> + <td>exempt-throttle-time indicates the percentage of time spent in broker network and I/O threads to process requests + that are exempt from throttling.</td> + </tr> </tbody></table> <h4><a id="selector_monitoring" href="#selector_monitoring">Common monitoring metrics for producer/consumer/connect/streams</a></h4>
