On 4/7/19 10:44 PM, melank...@synergentl.com wrote:
On 2019/04/04 15:15:37, Josh Elser <els...@apache.org> wrote:
Looks like your RegionServer process might have died if you can't
connect to its RPC port.
Did you look in the RegionServer log for any mention of an ERROR or
FATAL log message?
On 4/4/19 8:20 AM, melank...@synergentl.com wrote:
I have installed Hadoop single node
http://intellitech.pro/tutorial-hadoop-first-lab/ and Hbase
http://intellitech.pro/hbase-installation-on-ubuntu/ successfully. I am using
a Java agent to connect to the Hbase. After a random time period Hbase stop
working and the java agent gives following error message.
Call exception, tries=7, retries=7, started=8321 ms ago, cancelled=false,
msg=Call to db-2.c.xxx-dev.internal/xx.xx.0.21:16201 failed on connection
exception:
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: db-2.c.xxx-dev.internal/xx.xx.0.21:16201, details=row
'xxx,00000000001:1553904000000,99999999999999' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=db-2.c.xxx-dev.internal,16201,1553683263844, seqNum=-1
Here are the Hbase and zookeeper logs
hbase-hduser-regionserver-db-2.log
[main] zookeeper.ZooKeeperMain: Processing delete 2019-03-30 02:11:44,089 DEBUG
[main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Reading reply
sessionid:0x169bd98c099006e, packet:: clientPath:null serverPath:null
finished:false header:: 1,2 replyHeader:: 1,300964,0 request::
'/hbase/rs/db-2.c.stl-cardio-dev.internal%2C16201%2C1553683263844,-1 response::
null
hbase-hduser-zookeeper-db-2.log
server.FinalRequestProcessor: sessionid:0x169bd98c099004a type:getChildren
cxid:0x28e3ad zxid:0xfffffffffffffffe txntype:unknown reqpath:/hbase/splitWAL
my hbase-site.xml file is as follows
<configuration>
//Here you have to set the path where you want HBase to store its files.
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
//Here you have to set the path where you want HBase to store its built in
zookeeper files.
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>${hbase.tmp.dir}/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>
when I restart the Hbase it will start working again and stop working after few
days. I am wondering what would be the fix for this.
Thanks.
BR,
Melanka
Hi Josh,
Sorry for the late reply. I restarted the Hbase on 05/04/2019 and it was again
down on 06/04/2019 at 00.06 AM.
Log from hbase-root-regionserver-db-2 is as follows.
2019-04-04 04:42:26,047 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169d86a879b00bf, packet::
clientPath:null serverPath:null finished:false header:: 67,2 replyHeader::
67,776370,0 request::
'/hbase/rs/db-2.c.stl-cardio-dev.internal%2C16201%2C1554352093266,-1
response:: null
2019-04-04 04:42:26,047 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher:
regionserver:16201-0x169d86a879b00bf, quorum=localhost:2181, baseZNode=/hbase
Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected,
path=/hbase/rs/db-2.c.stl-cardio-dev.internal,16201,1554352093266
2019-04-04 04:42:26,047 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher:
regionserver:16201-0x169d86a879b00bf, quorum=localhost:2181, baseZNode=/hbase
Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected,
path=/hbase/rs
2019-04-04 04:42:26,050 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xxx.0.21:16201] zookeeper.ZooKeeper:
Closing session: 0x169d86a879b00bf
2019-04-04 04:42:26,050 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xx.0.21:16201] zookeeper.ClientCnxn:
Closing client for session: 0x169d86a879b00bf
2019-04-04 04:42:26,056 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169d86a879b00bf, packet::
clientPath:null serverPath:null finished:false header:: 68,-11 replyHeader::
68,776371,0 request:: null response:: null
2019-04-04 04:42:26,056 DEBUG
[regionserver/db-2.c.xxx-dev.internal/xx.xxx.0.21:16201] zookeeper.ClientCnxn:
Disconnecting client for session: 0x169d86a879b00bf
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.xxx.0.21:16201] zookeeper.ZooKeeper:
Session: 0x169d86a879b00bf closed
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.xxx.0.21:16201]
regionserver.HRegionServer: stopping server
db-2.c.xxx-dev.internal,16201,1554352093266; zookeeper connection closed.
2019-04-04 04:42:26,056 INFO
[regionserver/db-2.c.xxx-dev.internal/xxx.0.21:16201]
regionserver.HRegionServer: regionserver/db-2.c.xxx-dev.internal/xxx.0.21:16201
exiting
2019-04-04 04:42:26,056 ERROR [main] regionserver.HRegionServerCommandLine:
Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at
org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2831)
2019-04-04 04:42:26,057 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down for session: 0x169d86a879b00bf
2019-04-04 04:42:26,063 INFO [Thread-5] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35a9782c
2019-04-04 04:42:26,067 INFO [Thread-5] regionserver.ShutdownHook: Starting fs
shutdown hook thread.
2019-04-04 04:42:26,073 INFO [Thread-5] regionserver.ShutdownHook: Shutdown
hook finished.
Log from the hbase-hduser-regionserver-db-2 is ass follows.
2019-04-05 13:48:14,734 DEBUG [main] zookeeper.ZooKeeperMain: Processing delete
2019-04-05 13:48:14,754 DEBUG [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Reading reply sessionid:0x169eb8b7c230010, packet::
clientPath:null serverPath:null finished:false header:: 1,2 replyHeader::
1,783888,0 request::
'/hbase/rs/db-2.c.xxx-dev.internal%2C16201%2C1554434982329,-1 response:: null
Look harder, specifically for a FATAL message. `grep` is your friend.
