This may be related to :
https://issues.apache.org/jira/browse/HBASE-8912
It has started when I tried to install and run YCSB. I have created 'usertable'
and then tried to modify it couple times (added COMPRESSION),
HBase (0.94.6) stopped working (Master could not finish initialization)
I stopped the cluster and physically removed /hbase/usertable directory as well
as all ZK local stores. Restarted. No success.
I manually ran OfflineMetaRepair. Restarted. No success. This is FATAL error
in Master's log file.
For some reason, OfflineMetaRepair did not fix missing 'usertable'.
Please, advise. This is a development cluster with a large volume of data.
2013-07-27 23:08:56,504 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
The znode of region
TMO_NOV_INDEX-UPLOADS,38,1360181215845.2553b53773e3cb9030c3248768a3b0ca. has
been deleted.
2013-07-27 23:08:56,504 INFO org.apache.hadoop.hbase.master.AssignmentManager:
The master has opened the region
TMO_NOV_INDEX-UPLOADS,38,1360181215845.2553b53773e3cb9030c3248768a3b0ca. that
was online on sjc1-eng-perf-g1-grid06.carrieriq.com,60020,1374966494222
2013-07-27 23:08:56,504 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state : usertable,,1374962208806.249881162b6ad6d084b30507283f98b8.
state=PENDING_OPEN, ts=1374966536502,
server=sjc1-eng-perf-g1-grid14.carrieriq.com,60020,1374966494232 .. Cannot
transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
usertable,,1374962208806.249881162b6ad6d084b30507283f98b8. state=PENDING_OPEN,
ts=1374966536502,
server=sjc1-eng-perf-g1-grid14.carrieriq.com,60020,1374966494232 .. Cannot
transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-07-27 23:08:56,504 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
The znode of region
TMO_NOV_INDEX-UPLOADS,46,1360181215846.6f2a2eb3924ba5cb6ed22f966e6356e8. has
been deleted.
2013-07-27 23:08:56,505 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [email protected]
________________________________________
From: stack (JIRA) [[email protected]]
Sent: Saturday, July 27, 2013 3:21 PM
To: [email protected]
Subject: [jira] [Created] (HBASE-9063)
TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState
fails
stack created HBASE-9063:
----------------------------
Summary:
TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState
fails
Key: HBASE-9063
URL: https://issues.apache.org/jira/browse/HBASE-9063
Project: HBase
Issue Type: Bug
Components: test
Reporter: stack
Assignee: Jimmy Xiang
https://builds.apache.org/job/hbase-0.95-on-hadoop2/200/testReport/org.apache.hadoop.hbase.master/TestAssignmentManagerOnCluster/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/
{code}java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:1314)
at
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState(TestAssignmentManagerOnCluster.java:482)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){code}
Hope you don't mind my assigning it to you Jimmy. Thought you might be
interested.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Confidentiality Notice: The information contained in this message, including
any attachments hereto, may be confidential and is intended to be read only by
the individual or entity to whom this message is addressed. If the reader of
this message is not the intended recipient or an agent or designee of the
intended recipient, please note that any review, use, disclosure or
distribution of this message or its attachments, in any form, is strictly
prohibited. If you have received this message in error, please immediately
notify the sender and/or [email protected] and delete or destroy any
copy of this message and its attachments.
