[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-9393: --- Fix Version/s: 1.1.12 1.2.7 1.3.2 > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-9393-branch-1.patch, HBASE-9393.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v16.patch, HBASE-9393.v16.patch, > HBASE-9393.v17.patch, HBASE-9393.v18.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Release Note: To handle this issue client need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in only those versions. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-9393-branch-1.patch, HBASE-9393.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v16.patch, HBASE-9393.v16.patch, > HBASE-9393.v17.patch, HBASE-9393.v18.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9393: -- Resolution: Fixed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) Pushed to master and branch-1. [~busbey] interested in this for 1.2? [~mantonov] want this in 1.3? [~ndimiduk] maybe for 1.1? > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-9393-branch-1.patch, HBASE-9393.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v16.patch, HBASE-9393.v16.patch, > HBASE-9393.v17.patch, HBASE-9393.v18.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393-branch-1.patch Patch for branch-1 > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393-branch-1.patch, HBASE-9393.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v16.patch, HBASE-9393.v16.patch, > HBASE-9393.v17.patch, HBASE-9393.v18.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v18.patch Attached patch fixing the compilation error. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v17.patch, > HBASE-9393.v18.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v17.patch Attached the rebased patch with Javadoc > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v17.patch, > HBASE-9393.v1.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v16.patch Attaching the same patch again to trigger the QA. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Status: Patch Available (was: Open) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2, 1.0.1.1, 0.98.0, 0.94.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Status: Open (was: Patch Available) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2, 1.0.1.1, 0.98.0, 0.94.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v16.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, > HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, > HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, > HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lv9 updated HBASE-9393: --- Affects Version/s: 1.0.1.1 1.1.2 > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v15.patch Reattached for another QA run. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v15.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v15.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v15.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v14.patch Please review patch v14 and let me know your comments. Thanks. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v14.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v13.patch Patch addressing comments. Please review. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v13.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v12.patch Patch addressing Ted's comment. Please review. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v12.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v11.patch Fixed the last findbugs warning. Please review. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v11.patch, HBASE-9393.v2.patch, > HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, > HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: 0001-HBASE-9393.v10.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v10.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: (was: 0001-HBASE-9393.v10.patch) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: (was: HBASE-9393.v10.patch) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v10.patch Addressed findbugs warnings. Please review. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v10.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, > HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, > HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v9.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v8.patch Patch addressing check style warning and compilation error with pre Hadoop 2.6.4 and 2.7.1 versions. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch, HBASE-9393.v8.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v7.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: (was: HBASE-9393.v7.patch) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v7.patch Attached patch v7 after HADOOP-12805 is committed. Please review and let me know your thoughts. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Release Note: (was: To handle this issue user need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in those versions only. So from HBase side we have fixed this issue only in 2.0.0 version as by default the Hadoop version used there is 2.7.1. If a user is using Hadoop 2.6.4 or Hadoop 2.7.0+ version then they can back port this issue in their version to handle this issue.) Status: Patch Available (was: Reopened) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0, 0.94.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, > HBASE-9393.v7.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9393: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master branch. Thanks for the patch [~ashish singhi]. Sorry for delay in review. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Release Note: To handle this issue user need to have Hadoop client 2.6.4 or 2.7.0+ Hadoop version as CanUnBuffer interface which was added as part of HDFS-7694 is available in those versions only. So from HBase side we have fixed this issue only in 2.0.0 version as by default the Hadoop version used there is 2.7.1. If a user is using Hadoop 2.6.4 or Hadoop 2.7.0+ version then they can back port this issue in their version to handle this issue. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v6.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v6.patch Retry... > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v6.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, > HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v5.patch Addressed the comment. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9393: - Attachment: HBASE-9393.v5.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-9393: - Attachment: HBASE-9393.v5.patch Retry > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, > HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9393: -- Fix Version/s: 2.0.0 > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v3.patch Attached patch addressing review comment. Thanks for all the offline discussion on this, Anoop & Ram. For now this issue will be fixed only for 2.0.0. Once we plan to up our hadoop version to 2.7.x+ in any our branch code we can fix the issue there also as part of a back port jira. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v4.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v2.patch Patch addressing test failures and code refactor comment. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.v1.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Priority: Critical (was: Major) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi >Priority: Critical > Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, > HBASE-9393.v2.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Attachment: HBASE-9393.patch > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi > Attachments: HBASE-9393.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-9393: - Status: Patch Available (was: Reopened) > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0, 0.94.2 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya >Assignee: Ashish Singhi > Attachments: HBASE-9393.patch > > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] steven xu updated HBASE-9393: - Affects Version/s: 0.98.0 Hbase does not closing a closed socket resulting in many CLOSE_WAIT Key: HBASE-9393 URL: https://issues.apache.org/jira/browse/HBASE-9393 Project: HBase Issue Type: Bug Affects Versions: 0.94.2, 0.98.0 Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions Reporter: Avi Zrachya HBase dose not close a dead connection with the datanode. This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port. The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l 13156 [root@hd2-region3 ~]# ps -ef |grep 21592 root 17255 17219 0 12:26 pts/000:00:00 grep 21592 hbase21592 1 17 Aug29 ?03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-9393: -- Summary: Hbase does not closing a closed socket resulting in many CLOSE_WAIT (was: Hbase dose not closing a closed socket resulting in many CLOSE_WAIT ) Hbase does not closing a closed socket resulting in many CLOSE_WAIT Key: HBASE-9393 URL: https://issues.apache.org/jira/browse/HBASE-9393 Project: HBase Issue Type: Bug Affects Versions: 0.94.2 Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions Reporter: Avi Zrachya HBase dose not close a dead connection with the datanode. This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port. The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l 13156 [root@hd2-region3 ~]# ps -ef |grep 21592 root 17255 17219 0 12:26 pts/000:00:00 grep 21592 hbase21592 1 17 Aug29 ?03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.1#6144)
