[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593663#comment-14593663
]
Stephen Yuan Jiang commented on HBASE-13935:
--------------------------------------------
[~mbertozzi] I found the log after applying a similar fix in Branch-1.1 and
resolve the issue (not sure how we reached to this state with PV2, but the
patch fixed the issue and no inconsistency (note: the table is indeed in
ENABLED state when this issue happened):
{noformat}
2015-05-30 16:56:40,213 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION
2015-05-30 16:56:40,373 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_WRITE_FS_LAYOUT
2015-05-30 16:56:40,571 INFO [RegionOpenAndInitThread-hbase:namespace-1]
regionserver.HRegion: creating HRegion hbase:namespace HTD ==
'hbase:namespace', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER
=> 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10',
TTL => 'FOREVER', MIN_VERSIONS => '0', CACHE_DATA_IN_L1 => 'true',
KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '8192', IN_MEMORY => 'true',
BLOCKCACHE => 'true'} RootDir =
hdfs://ip-172-31-46-122.ec2.internal:8020/apps/hbase/data/.tmp Table name ==
hbase:namespace
2015-05-30 16:56:40,803 INFO [RegionOpenAndInitThread-hbase:namespace-1]
regionserver.HRegion: Closed
hbase:namespace,,1433004999998.9df720124fbb66974e95de65bddaee82.
2015-05-30 16:56:40,925 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_ADD_TO_META
2015-05-30 16:56:41,370 INFO [ProcedureExecutorThread-0]
hbase.MetaTableAccessor: Added 2
2015-05-30 16:56:41,480 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_ASSIGN_REGIONS
2015-05-30 16:56:41,481 INFO [ProcedureExecutorThread-0]
zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLED
to ENABLING
2015-05-30 16:56:41,486 INFO [ProcedureExecutorThread-0]
master.AssignmentManager: Assigning 1 region(s) to
ip-666-77-88-999.ec2.internal,16020,1433004979527
2015-05-30 16:56:41,509 INFO [ProcedureExecutorThread-0] master.RegionStates:
Transition {9df720124fbb66974e95de65bddaee82 state=OFFLINE, ts=1433005001494,
server=null} to {9df720124fbb66974e95de65bddaee82 state=PENDING_OPEN,
ts=1433005001509, server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
2015-05-30 16:56:41,872 INFO [ProcedureExecutorThread-0]
zookeeper.ZKTableStateManager: Moving table hbase:namespace state from ENABLING
to ENABLED
2015-05-30 16:56:41,887 INFO [AM.ZK.Worker-pool2-t5] master.RegionStates:
Transition {9df720124fbb66974e95de65bddaee82 state=PENDING_OPEN,
ts=1433005001509, server=ip-666-77-88-999.ec2.internal,16020,1433004979527} to
{9df720124fbb66974e95de65bddaee82 state=OPENING, ts=1433005001887,
server=ip-666-77-88-999.ec2.internal,16020,1433004979527}
2015-05-30 16:56:41,997 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_UPDATE_DESC_CACHE
2015-05-30 16:56:42,300 TRACE [ProcedureExecutorThread-0]
procedure.CreateTableProcedure: CreateTableProcedure (table=hbase:namespace)
user=hbase/[email protected] (auth:KERBEROS) id=9
state=RUNNABLE execute state=CREATE_TABLE_POST_OPERATION
{noformat}
> Orphaned namespace table ZK node should not prevent master to start
> -------------------------------------------------------------------
>
> Key: HBASE-13935
> URL: https://issues.apache.org/jira/browse/HBASE-13935
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 1.0.0, 0.98.13
> Reporter: Stephen Yuan Jiang
> Assignee: Stephen Yuan Jiang
> Fix For: 0.98.14, 1.0.2
>
> Attachments: HBASE-13935.v1-0.98.patch,
> HBASE-13935.v1-branch-1.0.patch
>
>
> Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
> older), we frequently see the following issue (orphaned ZK node) that prevent
> master to start (at least in testing):
> {noformat}
> 2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:60000] master.HMaster:
> Unhandled exception. Starting shutdown.
> org.apache.hadoop.hbase.TableExistsException: hbase:namespace
> at
> org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
> at java.lang.Thread.run(Thread.java:745)
> 2015-06-16 17:54:36,472 INFO [master:10.0.0.99:60000] master.HMaster:
> Aborting
> {noformat}
> The above call trace is from a 0.98.x test run. We saw similar issue in
> 1.0.x run, too.
> The proposed fix is to ignore the zk node and force namespace table creation
> to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
