[
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792938#comment-13792938
]
Colin Patrick McCabe commented on HBASE-9393:
---------------------------------------------
I guess I should also explain why this doesn't happen in branch-1 of Hadoop.
The reason is because Hadoop-1 had no socket cache and no grace period before
the sockets were closed. The client simply opened a new socket each time,
performed the op, and then closed it. This would result in (basically) no
sockets in {{CLOSE_WAIT}}. Remember {{CLOSE_WAIT}} only happens when the
server is waiting for the client to execute {{close}}.
Keeping sockets open is an optimization, but one that may require you to raise
your maximum number of file descriptors. If you are not happy with this
tradeoff, you can set {{dfs.client.socketcache.capacity}} to {{0}} and
{{dfs.datanode.socket.reuse.keepalive}} to {{0}} to get the old branch-1
behavior. It will be slower, though.
> Hbase dose not closing a closed socket resulting in many CLOSE_WAIT
> --------------------------------------------------------------------
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node,
> 7279 regions
> Reporter: Avi Zrachya
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect
> to the datanode because too many mapped sockets from one host to another on
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart
> hbase to solve the porblem, later in time it will incease to 60-100K sockets
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
> hbase 21592 1 17 Aug29 ? 03:29:06
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Dhbase.log.dir=/var/log/hbase
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...
--
This message was sent by Atlassian JIRA
(v6.1#6144)
