Re: forcing offline

Sun, 20 May 2012 09:04:06 -0700

Well, reading your hbase-master.log, it seems that Zookeeper is trying to access to several paths and they are not in the cluster: 
/hbase/rs
/hbase/unassigned

Zookeeper is running?
HBase is runnig?

Which version of Zookeeper and HBase are you using?


On 05/20/2012 11:02 AM, Michael Segel wrote:

What did you see when you ran the HBase shell's status?
Did you run status w higher details?
(see status help)


On May 20, 2012, at 2:12 AM, Ben Cuthbert wrote:


All

We run a load test and after about 3 hours our application stopped. Check the 
logs I see this in the hbase-master log

2012-05-20 08:08:17,251 INFO org.apache.hadoop.hbase.master.AssignmentManager: 
Region has been OFFLINE for too long, reassigning -ROOT-,,0.70236052 to a 
random server
2012-05-20 08:08:17,252 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Forcing OFFLINE; was=.META.,,1.1028785192 state=OFFLINE, ts=1337497517243
2012-05-20 08:08:17,252 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: 
Forcing OFFLINE; was=-ROOT-,,0.70236052 state=OFFLINE, ts=1337497517243
2012-05-20 08:10:10,309 INFO org.apache.zookeeper.server.NIOServerCnxn: 
Accepted socket connection from /0:0:0:0:0:0:0:1%0:62747
2012-05-20 08:10:10,315 INFO org.apache.zookeeper.server.NIOServerCnxn: Client 
attempting to establish new session at /0:0:0:0:0:0:0:1%0:62747
2012-05-20 08:10:10,316 INFO org.apache.zookeeper.server.NIOServerCnxn: 
Established session 0x137653a0e8e02fa with negotiated timeout 40000 for client 
/0:0:0:0:0:0:0:1%0:62747
2012-05-20 08:10:10,316 INFO org.apache.zookeeper.server.PrepRequestProcessor: 
Got user-level KeeperException when processing sessionid:0x137653a0e8e02fa 
type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error 
Path:/hbase Error:KeeperErrorCode = NodeExists for /hbase
2012-05-20 08:10:10,329 INFO org.apache.zookeeper.server.PrepRequestProcessor: 
Got user-level KeeperException when processing sessionid:0x137653a0e8e02fa 
type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error 
Path:/hbase/unassigned Error:KeeperErrorCode = NodeExists for /hbase/unassigned
2012-05-20 08:10:10,329 INFO org.apache.zookeeper.server.PrepRequestProcessor: 
Got user-level KeeperException when processing sessionid:0x137653a0e8e02fa 
type:create cxid:0x3 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error 
Path:/hbase/rs Error:KeeperErrorCode = NodeExists for /hbase/rs
2012-05-20 08:10:10,330 INFO org.apache.zookeeper.server.PrepRequestProcessor: 
Got user-level KeeperException when processing sessionid:0x137653a0e8e02fa 
type:create cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error 
Path:/hbase/table Error:KeeperErrorCode = NodeExists for /hbase/table


Hadoop seems to be up and running.

last log in the datanode is

12/05/20 06:15:25 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_-3639294708473848144_3329
12/05/20 06:26:20 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_2502932128500788221_3413
12/05/20 06:26:20 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_3390059684225099859_3440
12/05/20 06:59:32 INFO datanode.DataNode: BlockReport of 157 blocks took 19 
msec to generate and 3 msecs for RPC and NN processing
12/05/20 07:24:51 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_8954400942867609419_3363
12/05/20 07:55:51 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_-3650918785526360502_3387
12/05/20 07:59:33 INFO datanode.DataNode: BlockReport of 157 blocks took 20 
msec to generate and 3 msecs for RPC and NN processing
12/05/20 08:07:25 INFO datanode.DataBlockScanner: Verification succeeded for 
blk_786514597978592338_3336

I tried using hbase-explorer to view the tables but they all seem to down.


10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS 
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci


--
Marcos Luis Ortíz Valmaseda
 Data Engineer&&  Sr. System Administrator at UCI
 http://marcosluis2186.posterous.com
 http://www.linkedin.com/in/marcosluis2186
 Twitter: @marcosluis2186


10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS 
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci

Reply via email to