[jira] [Commented] (HBASE-25239) Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster “Failed to become active master”
[
https://issues.apache.org/jira/browse/HBASE-25239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237835#comment-17237835
]
Yongle Zhang commented on HBASE-25239:
--
[~anoop.hbase] Seems what you said is correct.
In 2.2, here is the code that waits for the NS to get online (*the while loop
waits for the NS*):
{code:java}
public void start() throws IOException {
if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
TableName.NAMESPACE_TABLE_NAME)) {
LOG.info("Namespace table not found. Creating...");
createNamespaceTable(masterServices);
}try {
// Wait for the namespace table to be initialized.
long startTime = EnvironmentEdgeManager.currentTime();
int timeout = conf.getInt(NS_INIT_TIMEOUT, DEFAULT_NS_INIT_TIMEOUT);
while (!isTableAvailableAndInitialized()) {
if (EnvironmentEdgeManager.currentTime() - startTime + 100 > timeout) {
// We can't do anything if ns is not online.
throw new IOException("Timedout " + timeout + "ms waiting for
namespace table to "
+ "be assigned and enabled: " + getTableState());
}
Thread.sleep(100);
}
} catch (InterruptedException e) {
throw (InterruptedIOException) new InterruptedIOException().initCause(e);
}
}
{code}
In master, the while loop that waits for the NS does not exist:
{code:java}
public void start() throws IOException {
TableState nsTableState =
MetaTableAccessor.getTableState(masterServices.getConnection(),
TableName.NAMESPACE_TABLE_NAME);
if (nsTableState != null && nsTableState.isEnabled()) {
migrateNamespaceTable();
}
loadNamespaceIntoCache();
}
private void migrateNamespaceTable() throws IOException {
try (Table nsTable =
masterServices.getConnection().getTable(TableName.NAMESPACE_TABLE_NAME);
ResultScanner scanner = nsTable.getScanner(
new
Scan().addFamily(TableDescriptorBuilder.NAMESPACE_FAMILY_INFO_BYTES).readAllVersions());
BufferedMutator mutator =
masterServices.getConnection().getBufferedMutator(TableName.META_TABLE_NAME)) {
for (Result result;;) {
result = scanner.next();
if (result == null) {
break;
}
Put put = new Put(result.getRow());
result
.getColumnCells(TableDescriptorBuilder.NAMESPACE_FAMILY_INFO_BYTES,
TableDescriptorBuilder.NAMESPACE_COL_DESC_BYTES)
.forEach(c -> put.addColumn(HConstants.NAMESPACE_FAMILY,
HConstants.NAMESPACE_COL_DESC_QUALIFIER, c.getTimestamp(),
CellUtil.cloneValue(c)));
mutator.mutate(put);
}
}
// schedule a disable procedure instead of block waiting here, as when
disabling a table we will
// wait until master is initialized, but we are part of the
initialization...
masterServices.getMasterProcedureExecutor().submitProcedure(
new
DisableTableProcedure(masterServices.getMasterProcedureExecutor().getEnvironment(),
TableName.NAMESPACE_TABLE_NAME, false));
}
{code}
> Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster
> “Failed to become active master”
>
>
> Key: HBASE-25239
> URL: https://issues.apache.org/jira/browse/HBASE-25239
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.3.3
>Reporter: Zhuqi Jin
>Priority: Major
>
> When we upgraded HBASE cluster from 2.2.0/2.3.3 to
> master(c303f9d329d578d31140e507bdbcbe3aa097042b), the HMaster on upgraded
> node failed to start.
> The error message is shown below:
> {code:java}
> 2020-11-03 02:52:27,809 ERROR [master/65cddff041f6:16000:becomeActiveMaster]
> master.HMaster: Failed to become active
> masterjava.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> or
[jira] [Commented] (HBASE-25239) Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster “Failed to become active master”
[
https://issues.apache.org/jira/browse/HBASE-25239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17227348#comment-17227348
]
Anoop Sam John commented on HBASE-25239:
Seems TableNamespaceManager.migrateNamespaceTable is not really waiting for the
NS table to get online? Did not read code.
> Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster
> “Failed to become active master”
>
>
> Key: HBASE-25239
> URL: https://issues.apache.org/jira/browse/HBASE-25239
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.3.3
>Reporter: Zhuqi Jin
>Priority: Major
>
> When we upgraded HBASE cluster from 2.2.0/2.3.3 to
> master(c303f9d329d578d31140e507bdbcbe3aa097042b), the HMaster on upgraded
> node failed to start.
> The error message is shown below:
> {code:java}
> 2020-11-03 02:52:27,809 ERROR [master/65cddff041f6:16000:becomeActiveMaster]
> master.HMaster: Failed to become active
> masterjava.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123)at
>
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360)...
> 4 more2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: Master server
> abort: loaded coprocessors are: []2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: * ABORTING
> master 65cddff041f6,16000,1604371935915: Unhandled exception. Starting
> shutdown. *java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.start
[jira] [Commented] (HBASE-25239) Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster “Failed to become active master”
[
https://issues.apache.org/jira/browse/HBASE-25239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17226998#comment-17226998
]
Zhuqi Jin commented on HBASE-25239:
---
[~zhangduo]
I don't quite understand what "there should be live region servers to host
namespace table and meta table." mean.
Can't I shut down all nodes and then start them after upgrading to master?
> Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster
> “Failed to become active master”
>
>
> Key: HBASE-25239
> URL: https://issues.apache.org/jira/browse/HBASE-25239
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.3.3
>Reporter: Zhuqi Jin
>Priority: Major
>
> When we upgraded HBASE cluster from 2.2.0/2.3.3 to
> master(c303f9d329d578d31140e507bdbcbe3aa097042b), the HMaster on upgraded
> node failed to start.
> The error message is shown below:
> {code:java}
> 2020-11-03 02:52:27,809 ERROR [master/65cddff041f6:16000:becomeActiveMaster]
> master.HMaster: Failed to become active
> masterjava.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123)at
>
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360)...
> 4 more2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: Master server
> abort: loaded coprocessors are: []2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: * ABORTING
> master 65cddff041f6,16000,1604371935915: Unhandled exception. Starting
> shutdown. *java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)a
[jira] [Commented] (HBASE-25239) Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster “Failed to become active master”
[
https://issues.apache.org/jira/browse/HBASE-25239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225179#comment-17225179
]
Duo Zhang commented on HBASE-25239:
---
We will migrate the data in namespace table to meta table, so there should be
live region servers to host namespace table and meta table.
> Upgrading HBase from 2.2.0/2.3.3 to master(3.0.0) fails because HMaster
> “Failed to become active master”
>
>
> Key: HBASE-25239
> URL: https://issues.apache.org/jira/browse/HBASE-25239
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.3.3
>Reporter: Zhuqi Jin
>Priority: Major
>
> When we upgraded HBASE cluster from 2.2.0/2.3.3 to
> master(c303f9d329d578d31140e507bdbcbe3aa097042b), the HMaster on upgraded
> node failed to start.
> The error message is shown below:
> {code:java}
> 2020-11-03 02:52:27,809 ERROR [master/65cddff041f6:16000:becomeActiveMaster]
> master.HMaster: Failed to become active
> masterjava.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123)at
>
> org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360)...
> 4 more2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: Master server
> abort: loaded coprocessors are: []2020-11-03 02:52:27,810 ERROR
> [master/65cddff041f6:16000:becomeActiveMaster] master.HMaster: * ABORTING
> master 65cddff041f6,16000,1604371935915: Unhandled exception. Starting
> shutdown. *java.lang.IllegalStateException: Expected the service
> ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at
>
> org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at
>
> org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362)at
>
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137)at
>
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626)at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2
> actions: RetriesExhaustedException: 2 times, servers with issues:at
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122)at
>
> org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93)at
>
> org.apache.hadoop.hbase.master.TableNamespaceManager.sta
