[
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112957#comment-15112957
]
Ashish Singhi commented on HBASE-9393:
--------------------------------------
Ran some tests,
0. randomWrite
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=134
--rows=100000 randomWrite 20
{noformat}
1. randomRead
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000
randomRead 20
{noformat}
Without patch
{noformat}
2016-01-22 23:49:52,305 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Summary of timings (ms): [7223, 7034, 6950, 6945, 6882, 6830,
7107, 7097, 7122, 6675, 7072, 6636, 7080, 6533, 6987, 6305, 7227, 7172, 6608,
6589]
2016-01-22 23:49:52,306 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Min: 6305ms Max: 7227ms Avg: 6903ms
{noformat}
With patch
{noformat}
2016-01-22 23:43:08,695 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Summary of timings (ms): [6406, 6648, 7623, 6678, 7163, 6673,
7150, 6712, 6412, 7169, 6364, 6214, 7293, 7484, 7633, 7212, 7350, 6447, 7101,
6499]
2016-01-22 23:43:08,696 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Min: 6214ms Max: 7633ms Avg: 6911ms
{noformat}
2. sequentialRead
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000
sequentialRead 20
{noformat}
Without patch
{noformat}
2016-01-22 23:54:56,024 INFO [main] hbase.PerformanceEvaluation:
[SequentialReadTest] Summary of timings (ms): [6476, 6150, 6291, 6381, 6284,
6182, 6069, 6350, 6394, 6200, 6260, 6349, 6240, 5974, 6014, 5965, 6483, 6025,
6098, 6389]
2016-01-22 23:54:56,025 INFO [main] hbase.PerformanceEvaluation:
[SequentialReadTest] Min: 5965ms Max: 6483ms Avg: 6228ms
{noformat}
With patch
{noformat}
2016-01-22 23:58:40,519 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Summary of timings (ms): [6985, 6720, 6970, 6756, 6468, 6890,
6719, 7003, 6348, 6803, 6584, 6846, 6793, 6496, 6490, 6879, 6450, 6663, 6921,
6896]
2016-01-22 23:58:40,520 INFO [main] hbase.PerformanceEvaluation:
[RandomReadTest] Min: 6348ms Max: 7003ms Avg: 6734ms
{noformat}
3. randomSeekScan
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000
randomSeekScan 20
{noformat}
Without patch
{noformat}
2016-01-23 00:12:01,954 INFO [main] hbase.PerformanceEvaluation:
[RandomSeekScanTest] Summary of timings (ms): [105473, 107416, 88796, 91563,
104032, 101899, 99557, 103790, 107929, 94213, 103251, 101177, 106168, 106903,
106086, 101905, 97543, 68672, 91004, 105064]
2016-01-23 00:12:01,954 INFO [main] hbase.PerformanceEvaluation:
[RandomSeekScanTest] Min: 68672ms Max: 107929ms Avg: 99622ms
{noformat}
With patch
{noformat}
2016-01-23 00:05:07,185 INFO [main] hbase.PerformanceEvaluation:
[RandomSeekScanTest] Summary of timings (ms): [78781, 82973, 76085, 81127,
74558, 74974, 60761, 77760, 80286, 70820, 71463, 74105, 70433, 64313, 80937,
82408, 81356, 83155, 65988, 82360]
2016-01-23 00:05:07,186 INFO [main] hbase.PerformanceEvaluation:
[RandomSeekScanTest] Min: 60761ms Max: 83155ms Avg: 75732ms
{noformat}
4. filterScan
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=100
filterScan 3
{noformat}
Without patch
{noformat}
2016-01-23 01:01:10,168 INFO [main] hbase.PerformanceEvaluation:
[FilteredScanTest] Summary of timings (ms): [417507, 425604, 420263]
2016-01-23 01:01:10,169 INFO [main] hbase.PerformanceEvaluation:
[FilteredScanTest] Min: 417507ms Max: 425604ms Avg: 421124ms
{noformat}
With patch
{noformat}
2016-01-23 01:17:28,614 INFO [main] hbase.PerformanceEvaluation:
[FilteredScanTest] Summary of timings (ms): [359967, 358833, 359256]
2016-01-23 01:17:28,615 INFO [main] hbase.PerformanceEvaluation:
[FilteredScanTest] Min: 358833ms Max: 359967ms Avg: 359352ms
{noformat}
> Hbase does not closing a closed socket resulting in many CLOSE_WAIT
> --------------------------------------------------------------------
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node,
> 7279 regions
> Reporter: Avi Zrachya
> Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch,
> HBASE-9393.v2.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect
> to the datanode because too many mapped sockets from one host to another on
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart
> hbase to solve the porblem, later in time it will incease to 60-100K sockets
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219 0 12:26 pts/0 00:00:00 grep 21592
> hbase 21592 1 17 Aug29 ? 03:29:06
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Dhbase.log.dir=/var/log/hbase
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
