Hi, all,
when I configure ipc pool in hbase-site.xml as follows:
<property>
<name>hbase.client.ipc.pool.type</name>
<value>RoundRobinPool</value>
</property>
<property>
<name>hbase.client.ipc.pool.size</name>
<value>1</value>
</property>
the start and stop processes of hbase run without any problem.
But if I set hbase.client.ipc.pool.size to 10 (20, 30, or possibly some
other values), some problems occur.
In the regionserver's log, I found that the regionserver continuously
attempted connect to master server:
2013-06-24 16:59:11,734 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:14,738 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:17,741 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:20,744 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
......many more
About 30 minutes after the master started, in the master's log,
client.ServerCallable began to throw the following exception repeatedly:
2013-06-24 17:32:47,233 WARN
[HOSTNAME-TEST,60000,1372064256847-CatalogJanitor] client.ServerCallable:
Call exception, tries=0, numRetries=200
java.io.IOException: Call to HOSTNAME-TEST/192.168.1.35:60020 failed on
local exception: java.io.IOException: Call 31 not added as the
connection HOSTNAME-TEST/
192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000 is closing
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1368)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1340)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1540)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1597)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:21367)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:290)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:147)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:55)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:174)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:216)
at
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:131)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:585)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:164)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:102)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:80)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:165)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:117)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:219)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:99)
at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Call 31 not added as the
connection
HOSTNAME-TEST/192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000
is closing
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.addCall(RpcClient.java:469)
at
org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1413)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1314)
... 19 more
So I want to know what is going wrong?
Extra information:
hbase issue: https://issues.apache.org/jira/browse/HBASE-8790
environment: CentOS 5.9 x86_64, java version "1.6.0_45", CDH4.3
The hbase cluster is a fresh start with only one regionserver.
----------------------------
Best Wishes
