[
https://issues.apache.org/jira/browse/HBASE-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410269#comment-13410269
]
Lars George commented on HBASE-6294:
------------------------------------
Another issue reported by someone on IM is that an entry in /hbase/tables is
causing an problem where you cannot create a table with a previously known
name. For some reason the table was first disabled, then HDFS wiped clean, yet
the entry in ZK remains and causes some check to fail when you try to create
the table.
> Detect leftover data in ZK after a user delete all its HBase data
> -----------------------------------------------------------------
>
> Key: HBASE-6294
> URL: https://issues.apache.org/jira/browse/HBASE-6294
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.96.0, 0.94.2
>
>
> It seems we have a new failure mode when a user deletes the hbase root.dir
> but doesn't delete the ZK data. For example a user on IRC came with this log:
> {noformat}
> 2012-06-30 09:07:48,017 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
> region: kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.
> 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors:
> The following folder is in HBase's root directory and doesn't contain a table
> descriptor, do consider deleting it: kw
> 2012-06-30 09:07:48,018 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:34193-0x1383bfe01b70001 Attempting to transition node
> 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,018 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694,
> region=2e8a318837602c9c9961e9d690b7fd02
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_FAILED_OPEN, server=localhost,34193,1341036300138,
> region=b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
> event for b254af24c9127b8bb22cb6d24e523dad
> 2012-06-30 09:07:48,020 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE;
> was=kw_r,,1340981822374.b254af24c9127b8bb22cb6d24e523dad. state=CLOSED,
> ts=1341036467998, server=localhost,34193,1341036300138
> 2012-06-30 09:07:48,020 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50890-0x1383bfe01b70000 Creating (or updating) unassigned node for
> b254af24c9127b8bb22cb6d24e523dad with OFFLINE state
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:34193-0x1383bfe01b70001 Successfully transitioned node
> 2e8a318837602c9c9961e9d690b7fd02 from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING
> 2012-06-30 09:07:48,028 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Opening region: {NAME =>
> 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.', STARTKEY => '', ENDKEY
> => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,}
> 2012-06-30 09:07:48,029 ERROR
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
> of region=kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02., starting to
> roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3490)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3628)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3487)
> ... 7 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:133)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:125)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:411)
> ... 11 more
> 2012-06-30 09:07:48,031 INFO
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of
> region {NAME => 'kw,,1340981821308.2e8a318837602c9c9961e9d690b7fd02.',
> STARTKEY => '', ENDKEY => '', ENCODED => 2e8a318837602c9c9961e9d690b7fd02,}
> failed, marking as FAILED_OPEN in ZK
> 2012-06-30 09:07:48,032 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:34193-0x1383bfe01b70001 Attempting to transition node
> 2e8a318837602c9c9961e9d690b7fd02 from RS_ZK_REGION_OPENING to
> RS_ZK_REGION_FAILED_OPEN
> 2012-06-30 09:07:48,031 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING, server=localhost,34193,1341036300138,
> region=2e8a318837602c9c9961e9d690b7fd02
> 2012-06-30 09:07:48,043 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=M_ZK_REGION_OFFLINE, server=localhost,50890,1341036299694,
> region=b254af24c9127b8bb22cb6d24e523dad
> {noformat}
> The exception itself is not very useful, nor is the NPE deep in the coproc
> stack. What was really useful was this:
> {noformat}
> 2012-06-30 09:07:48,017 WARN org.apache.hadoop.hbase.util.FSTableDescriptors:
> The following folder is in HBase's root directory and doesn't contain a table
> descriptor, do consider deleting it: kw
> {noformat}
> So the HBase wants to assign a region from a table that doesn't exist and we
> fail in an obscure way. I told the user to shut down HBase, nuke
> /tmp/hbase-user as it will contain both the HBase data and the ZK data, and
> restart. It worked.
> This situation is new in 0.94, we need to detect it so our users have a
> better experience getting started with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
