Zhuqi Jin created HBASE-24557:
---------------------------------
Summary: HBase upgrade from 2.3 to master fails because
master.HMaster fails to become active master
Key: HBASE-24557
URL: https://issues.apache.org/jira/browse/HBASE-24557
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: master
Reporter: Zhuqi Jin
Attachments: hbase--master-085bbffdc52f.log
When we tested upgrading HBase from 2.3 to
master(c303f9d329d578d31140e507bdbcbe3aa097042b), the upgraded node failed to
start.
The error message is shown below:
{code:java}
2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster]
master.HMaster: Failed to become active master2020-06-13 20:23:26,409 ERROR
[master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become
active masterjava.lang.IllegalStateException: Expected the service
ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
actions: RetriesExhaustedException: 2 times, servers with issues: at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360)
... 4 more2020-06-13 20:23:26,410 ERROR
[master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Master server
abort: loaded coprocessors are:
[org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]2020-06-13
20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster]
master.HMaster: ***** ABORTING master 085bbffdc52f,16000,1592079787433:
Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException:
Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the
service has FAILED at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
actions: RetriesExhaustedException: 2 times, servers with issues: at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123)
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249)
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360)
... 4 more
{code}
It can be reproduced through the following steps:
* Start a single-node HBase 2.3 with the following configuration.
{code:java}
<property>
<name>hbase.rootdir</name>
<value>/var/lib/hbase</value>
</property> <property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/var/lib/zookeeper</value>
</property>{code}
* Use HBase performance evaluation tool to generate data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
* Upgrade the node to master with the same configuration. The new version node
failed to start. And there is error message in /hbase/logs.
When I used the default configuration, the error message is the same as before.
I also attached the entire log file. [^hbase--master-085bbffdc52f.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
