[
https://issues.apache.org/jira/browse/HBASE-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699457#comment-17699457
]
Rajeshbabu Chintaguntla edited comment on HBASE-27698 at 3/13/23 6:19 AM:
--------------------------------------------------------------------------
[~vjasani]
We are getting following exception when we do express upgrade from HBase 1.1.2
to 2.5.2 which expects zookeeper znode to be present to create the master data
for meta and region info in master cache.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
was (Author: rajeshbabu):
[~vjasani]
We are getting following exception when we do express upgrade which expects
zookeeper znode to be created.
{noformat}
2023-03-10 09:49:10,828 WARN [PEWorker-3] procedure.InitMetaProcedure: Failed
to init meta, suspend 1000secs
java.io.IOException: Meta table is not partial, please sideline this meta
directory or run HBCK to fix this meta table, e.g. rebuild the server hostname
in ZNode for the meta region
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.deleteMetaTableDirectoryIfPartial(InitMetaProcedure.java:199)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.writeFsLayout(InitMetaProcedure.java:78)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.InitMetaProcedure.executeFromState(InitMetaProcedure.java:54)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1394)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1960)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987)
{noformat}
As of now there is no way or tool to create the meta location znode.
[~nihaljain.cs] created a script to create the zookeeper node and able to
perform the migration. He is going to contribute it.
Or else other way to migrate is rebuild the meta using HBCK2 which will be
clumsy as we need to perform lot of steps to adding missing regions in meta,
run assignments or restart the services etc.
> Migrate meta locations from zookeeper to master data may not always possible
> if we migrate from 1.x HBase
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-27698
> URL: https://issues.apache.org/jira/browse/HBASE-27698
> Project: HBase
> Issue Type: Bug
> Components: migration
> Affects Versions: 2.5.0
> Reporter: Rajeshbabu Chintaguntla
> Assignee: Rajeshbabu Chintaguntla
> Priority: Major
>
> In HBase 1.x versions meta server location from zookeeper will be removed
> when the server stopped. In such cases migrating to 2.5.x branches may not
> create any meta entries in master data. So in case if we could not find the
> meta location from zookeeper we can get meta location from wal directories
> with .meta extension and add to master data.
> {noformat}
> private void tryMigrateMetaLocationsFromZooKeeper() throws IOException,
> KeeperException {
> // try migrate data from zookeeper
> try (ResultScanner scanner =
> masterRegion.getScanner(new
> Scan().addFamily(HConstants.CATALOG_FAMILY))) {
> if (scanner.next() != null) {
> // notice that all replicas for a region are in the same row, so the
> migration can be
> // done with in a one row put, which means if we have data in catalog
> family then we can
> // make sure that the migration is done.
> LOG.info("The {} family in master local region already has data in
> it, skip migrating...",
> HConstants.CATALOG_FAMILY_STR);
> return;
> }
> }
> // start migrating
> byte[] row =
> CatalogFamilyFormat.getMetaKeyForRegion(RegionInfoBuilder.FIRST_META_REGIONINFO);
> Put put = new Put(row);
> List<String> metaReplicaNodes = zooKeeper.getMetaReplicaNodes();
> StringBuilder info = new StringBuilder("Migrating meta locations:");
> for (String metaReplicaNode : metaReplicaNodes) {
> int replicaId =
> zooKeeper.getZNodePaths().getMetaReplicaIdFromZNode(metaReplicaNode);
> RegionState state = MetaTableLocator.getMetaRegionState(zooKeeper,
> replicaId);
> info.append(" ").append(state);
> put.setTimestamp(state.getStamp());
> MetaTableAccessor.addRegionInfo(put, state.getRegion());
> if (state.getServerName() != null) {
> MetaTableAccessor.addLocation(put, state.getServerName(),
> HConstants.NO_SEQNUM, replicaId);
> }
>
> put.add(CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY).setRow(put.getRow())
> .setFamily(HConstants.CATALOG_FAMILY)
>
> .setQualifier(RegionStateStore.getStateColumn(replicaId)).setTimestamp(put.getTimestamp())
>
> .setType(Cell.Type.Put).setValue(Bytes.toBytes(state.getState().name())).build());
> }
> if (!put.isEmpty()) {
> LOG.info(info.toString());
> masterRegion.update(r -> r.put(put));
> } else {
> LOG.info("No meta location available on zookeeper, skip migrating...");
> }
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
