[
https://issues.apache.org/jira/browse/ZOOKEEPER-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750165#comment-16750165
]
Hudson commented on ZOOKEEPER-3242:
-----------------------------------
SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #202 (See
[https://builds.apache.org/job/Zookeeper-trunk-single-thread/202/])
ZOOKEEPER-3242: Add server side connecting throttling (andor: rev
668828503389f3e542b846019bace8b9842fa428)
* (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java
* (add)
zookeeper-server/src/test/java/org/apache/zookeeper/server/BlueThrottleTest.java
* (edit)
zookeeper-server/src/test/java/org/apache/zookeeper/server/admin/CommandsTest.java
* (add)
zookeeper-server/src/main/java/org/apache/zookeeper/server/BlueThrottle.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java
* (add)
zookeeper-server/src/main/java/org/apache/zookeeper/server/ClientCnxnLimitException.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxnFactory.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerMXBean.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxnFactory.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxn.java
* (edit)
zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerBean.java
> Add server side connecting throttling
> -------------------------------------
>
> Key: ZOOKEEPER-3242
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3242
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Reporter: Jie Huang
> Assignee: Jie Huang
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> On-going performance investigation at Facebook has demonstrated that
> Zookeeper is easily overwhelmed by spikes in connection rates and/or write
> request rates. Zookeeper performance gets progressively worse, clients
> timeout and try to reconnect (exacerbating the problem) and things enter a
> death spiral. To solve this problem, we need to add load protection to
> Zookeeper via rate limiting and work shedding.
>
> This Jira adds a new connection rate limiting mechanism to Zookeeper in hopes
> of preventing Zookeeper from becoming overwhelmed during connection spikes.
> The new throttle is focused on limiting connections per second. The throttle
> is implemented as a token-bucket with optional probabilistic dropping based
> on the BLUE queue management algorithm.
>
> This token-bucket design allows the throttle to allow short bursts to pass,
> while still capping the total number of requests per second. However, an
> issue with a token bucket approach is that the wall clock arrival time of
> requests affects the probability of a request being allowed to pass or not.
> Under constant load this can lead to request starvation for requests that
> constantly arrive later than the majority. The optional probabilistic
> dropping mechanism is designed to combat this, making rejections a random
> event with little skew based on arrival time.
>
> A more verbose description can be found in the comments in
> org.apache.zookeeper.server.BlueThrottle.
>
> By default, both the token-bucket and probabilistic dropping mechanism are
> disabled. Enabling and tuning the throttles can be done both via Java system
> properties as well as against a running node via JMX.
>
> The throttle has been tested and benchmarked at Facebook.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
