Rest client does not close zookeeper sessions (leaking sessions for each GET or
PUT)
------------------------------------------------------------------------------------
Key: HBASE-4733
URL: https://issues.apache.org/jira/browse/HBASE-4733
Project: HBase
Issue Type: Bug
Components: rest
Affects Versions: 0.90.4
Environment: Fedora 13.
Reporter: jack levin
Fix For: 0.90.4
This will appear in the log once the zookeeper connection/session leaking will
grow to 2000, zookeeper won't be able to accept any more connections causing
REST RPC calls to fail, here is the log when the problem is in progress:
2011-10-26 01:35:49,270 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,300 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,317 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,321 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,323 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.101.255.3 - max is 2000
2011-10-26 01:35:49,367 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,375 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.4 - max is 2000
2011-10-26 01:35:49,382 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.2 - max is 2000
2011-10-26 01:35:49,404 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,429 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.2 - max is 2000
2011-10-26 01:35:49,439 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,469 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.3 - max is 2000
2011-10-26 01:35:49,489 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.101.255.3 - max is 2000
2011-10-26 01:35:49,501 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.101.255.2 - max is 2000
2011-10-26 01:35:49,584 WARN org.apache.zookeeper.server.NIOServerCnxn: Too
many connections from /10.102.255.2 - max is 2000
After the fix, the log looks much better, and we can observe zookeeper
connections closing after every RPC call:
2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn:
Accepted socket connection from /10.101.255.2:37225
2011-11-02 15:50:14,339 INFO org.apache.zookeeper.server.NIOServerCnxn: Client
attempting to establish new session at /10.101.255.2:37225
2011-11-02 15:50:14,340 INFO org.apache.zookeeper.server.NIOServerCnxn:
Established session 0x3352857cb1f19f with negotiated timeout 180000 for client
/10.101.255.2:37225
2011-11-02 15:50:14,363 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed
socket connection for client /10.101.255.2:37225 which had sessionid
0x3352857cb1f19f
2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn:
Accepted socket connection from /10.101.255.2:38089
2011-11-02 15:50:14,723 INFO org.apache.zookeeper.server.NIOServerCnxn: Client
attempting to establish new session at /10.101.255.2:38089
2011-11-02 15:50:14,725 INFO org.apache.zookeeper.server.NIOServerCnxn:
Established session 0x3352857cb1f1a0 with negotiated timeout 180000 for client
/10.101.255.2:38089
2011-11-02 15:50:14,771 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed
socket connection for client /10.101.255.2:38089 which had sessionid
0x3352857cb1f1a0
2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn:
Accepted socket connection from /10.101.255.3:34085
2011-11-02 15:50:16,326 INFO org.apache.zookeeper.server.NIOServerCnxn: Client
attempting to establish new session at /10.101.255.3:34085
2011-11-02 15:50:16,328 INFO org.apache.zookeeper.server.NIOServerCnxn:
Established session 0x3352857cb1f1a1 with negotiated timeout 180000 for client
/10.101.255.3:34085
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
