[
https://issues.apache.org/jira/browse/HBASE-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ke Han updated HBASE-28519:
---------------------------
Description:
h1. Reproduce
Start up HBase-2.5.8 cluster: 4 nodes: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
Execute one read command: LIST, then perform a full-stop upgrade to
HBase-3.0.0-beta-1 cluster: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
HMaster aborts with the following exception:
{code:java}
2024-04-13T03:47:15,969 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: Failed to become active master
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
... 5 more
2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: Master server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.quotas.MasterQuotasObserver]
2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: ***** ABORTING master hmaster,16000,1712980015693: Unhandled
exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
... 5 more
2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
master.HMaster: ***** STOPPING master 'hmaster,16000,1712980015693' *****
2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
master.HMaster: STOPPED: Stopped by master/hmaster:16000:becomeActiveMaster
2024-04-13T03:47:15,971 INFO [master/hmaster:16000] hbase.HBaseServerBase:
Stop info server {code}
I also tried upgrading from 2.5.8 to 3.0.0-alpha-1 and 3.0.0-alpha-4, they will
encounter the same exception.
I have attached my full logs.
h1. Root Cause
The exception is thrown when migrating Namespace data. The HBase doc mentions
"hbase:namespace table has been removed and fold into hbase:meta". However, the
exact root cause is still unclear.
was:
h1. Reproduce
Start up HBase-2.5.8 cluster: 4 nodes: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
* HDFS is running Hadoop-2.10.2
Execute one read command: LIST, then perform a full-stop upgrade to
HBase-3.0.0-beta-1 cluster: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
HMaster will abort with the following exception
{code:java}
2024-04-13T03:47:15,969 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: Failed to become active master
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
... 5 more
2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: Master server abort: loaded coprocessors are:
[org.apache.hadoop.hbase.quotas.MasterQuotasObserver]
2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
master.HMaster: ***** ABORTING master hmaster,16000,1712980015693: Unhandled
exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl
[FAILED] to be RUNNING, but the service has FAILED
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues:
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
... 5 more
2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
master.HMaster: ***** STOPPING master 'hmaster,16000,1712980015693' *****
2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
master.HMaster: STOPPED: Stopped by master/hmaster:16000:becomeActiveMaster
2024-04-13T03:47:15,971 INFO [master/hmaster:16000] hbase.HBaseServerBase:
Stop info server {code}
I also tried upgrading from 2.5.8 to 3.0.0-alpha-1 and 3.0.0-alpha-4, both will
have a similar exception.
h1. Root Cause
The exception is thrown when migrating Namespace data. The HBase doc mentions
"hbase:namespace table has been removed and fold into hbase:meta". However, the
exact root cause is still unclear.
> HMaster crash when upgrading from HBase-2.5.8 to HBase-3.0.0-beta-1
> -------------------------------------------------------------------
>
> Key: HBASE-28519
> URL: https://issues.apache.org/jira/browse/HBASE-28519
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha-1, 3.0.0-alpha-4, 3.0.0-beta-1, 2.5.8
> Reporter: Ke Han
> Priority: Major
> Attachments: hbase--master-64f850a4e287.log
>
>
> h1. Reproduce
> Start up HBase-2.5.8 cluster: 4 nodes: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
> Execute one read command: LIST, then perform a full-stop upgrade to
> HBase-3.0.0-beta-1 cluster: 1 HM, 2 RS, 1 HDFS (hadoop-2.10.2).
> HMaster aborts with the following exception:
> {code:java}
> 2024-04-13T03:47:15,969 ERROR [master/hmaster:16000:becomeActiveMaster]
> master.HMaster: Failed to become active master
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
> ~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
> at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> ... 5 more
> 2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.quotas.MasterQuotasObserver]
> 2024-04-13T03:47:15,970 ERROR [master/hmaster:16000:becomeActiveMaster]
> master.HMaster: ***** ABORTING master hmaster,16000,1712980015693: Unhandled
> exception. Starting shutdown. *****
> java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1535)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1204)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2494)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:613)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
> ~[hbase-common-3.0.0-beta-1.jar:3.0.0-beta-1]
> at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)
> ~[hbase-client-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:92)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:122)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> at
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
> ~[hbase-shaded-miscellaneous-4.1.5.jar:4.1.5]
> at
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1532)
> ~[hbase-server-3.0.0-beta-1.jar:3.0.0-beta-1]
> ... 5 more
> 2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
> master.HMaster: ***** STOPPING master 'hmaster,16000,1712980015693' *****
> 2024-04-13T03:47:15,971 INFO [master/hmaster:16000:becomeActiveMaster]
> master.HMaster: STOPPED: Stopped by master/hmaster:16000:becomeActiveMaster
> 2024-04-13T03:47:15,971 INFO [master/hmaster:16000] hbase.HBaseServerBase:
> Stop info server {code}
> I also tried upgrading from 2.5.8 to 3.0.0-alpha-1 and 3.0.0-alpha-4, they
> will encounter the same exception.
> I have attached my full logs.
> h1. Root Cause
> The exception is thrown when migrating Namespace data. The HBase doc mentions
> "hbase:namespace table has been removed and fold into hbase:meta". However,
> the exact root cause is still unclear.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
