Stop accepting connections when close to file descriptor limit
--------------------------------------------------------------

                 Key: ZOOKEEPER-759
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759
             Project: Zookeeper
          Issue Type: Improvement
          Components: server
            Reporter: Travis Crawford


Zookeeper always tries to accept new connections, throwing an exception if out 
of file descriptors. An improvement would be denying new client connections 
when close to the limit.

Additionally, file-descriptor limits+usage should be exported to the monitoring 
four-letter word, should that get implemented (see ZOOKEEPER-744).


DETAILS

A Zookeeper ensemble I administer recently suffered an outage when one node was 
restarted with the low system-default ulimit of 1024 file descriptors and later 
ran out. File descriptor usage+max are already being monitored by the following 
MBeans:

- java.lang.OperatingSystem.MaxFileDescriptorCount
- java.lang.OperatingSystem.OpenFileDescriptorCount

They're described (rather tersely) at:

http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html

This feature request is for the following:

(a) Stop accepting new connections when OpenFileDescriptorCount is close to 
MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be 
denied, logged to disk at debug level, and increment a 
``ConnectionDeniedCount`` MBean counter.

(b) Begin accepting new connections when usage drops below some configurable 
threshold, defaulting to 90% of FD usage, basically the high/low watermark 
model.

(c) Update the administrators guide with a comment about using an appropriate 
FD limit.

(d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for:

zookeeper_open_file_descriptor_count
zookeeper_max_file_descriptor_count
zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not 
all zk's have the same max FD value

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to