This is an automated email from the ASF dual-hosted git repository.
wchevreuil pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hbase-operator-tools.git
The following commit(s) were added to refs/heads/master by this push:
new 5220e87 HBASE-22567 [HBCK2] Add new methods for dealing with missing
regions in META while Master is online
5220e87 is described below
commit 5220e87171ccc7cded16b2d39d30dd19d763cf7f
Author: Wellington Chevreuil <[email protected]>
AuthorDate: Fri Sep 6 15:23:00 2019 +0100
HBASE-22567 [HBCK2] Add new methods for dealing with missing regions in
META while Master is online
Signed-off-by: stack <[email protected]>
---
hbase-hbck2/README.md | 110 ++++++++++-
hbase-hbck2/pom.xml | 6 +
.../org/apache/hbase/FsRegionsMetaRecoverer.java | 129 +++++++++++++
.../src/main/java/org/apache/hbase/HBCK2.java | 214 +++++++++++++++++++++
.../apache/hbase/TestFsRegionsMetaRecoverer.java | 209 ++++++++++++++++++++
.../src/test/java/org/apache/hbase/TestHBCK2.java | 147 ++++++++++++++
6 files changed, 813 insertions(+), 2 deletions(-)
diff --git a/hbase-hbck2/README.md b/hbase-hbck2/README.md
index 03491d2..fceebd5 100644
--- a/hbase-hbck2/README.md
+++ b/hbase-hbck2/README.md
@@ -77,6 +77,32 @@ Options:
-z,--zookeeper.znode.parent <arg> parent znode of hbase
ensemble
Command:
+ addFsRegionsMissingInMeta <NAMESPACE|NAMESPACE:TABLENAME>...
+ Options:
+ -d,--force_disable aborts fix for table if disable fails.
+ To be used in scenarios where some regions may be missing in META,
+ but there's still a valid 'regioninfo' metadata file on HDFS.
+ This is a lighter version of 'OfflineMetaRepair tool commonly used for
+ similar issues on 1.x release line.
+ This command needs META to be online. For each table name passed as
+ parameter, it performs a diff between regions available in META,
+ against existing regions dirs on HDFS. Then, for region dirs with
+ no matches in META, it reads regioninfo metadata file and
+ re-creates given region in META. Regions are re-created in 'CLOSED'
+ state at META table only, but not in Masters' cache, and are not
+ assigned either. To get these regions online, run HBCK2 'assigns'command
+ printed at the end of this command results for convenience.
+
+ NOTE: If using hbase releases older than 2.3.0, a rolling restart of
+ HMasters is needed prior to executing the provided 'assigns' command.
+
+ An example adding missing regions for tables 'tbl_1' on default
+ namespace, 'tbl_2' on namespace 'n1' and for all tables from
+ namespace 'n2':
+ $ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2
+ Returns HBCK2 'assigns' command with all re-inserted regions.
+ SEE ALSO: reportMissingRegionsInMeta
+
assigns [OPTIONS] <ENCODED_REGIONNAME>...
Options:
-o,--override override ownership by another procedure
@@ -109,6 +135,31 @@ Command:
checkup. Default checks all tables. Modified regions will need to be
reopened to pick-up changes.
+ reportMissingRegionsInMeta <NAMESPACE|NAMESPACE:TABLENAME>...
+ To be used in scenarios where some regions may be missing in META,
+ but there's still a valid 'regioninfo metadata file on HDFS.
+ This is a checking only method, designed for reporting purposes and
+ doesn't perform any fixes, providing a view of which regions (if any)
+ would get re-added to meta, grouped by respective table/namespace.
+ To effectively re-add regions in meta, addFsRegionsMissingInMeta should be
executed.
+ This command needs META to be online. For each namespace/table passed
+ as parameter, it performs a diff between regions available in META,
+ against existing regions dirs on HDFS. Region dirs with no matches
+ are printed grouped under its related table name. Tables with no
+ missing regions will show a 'no missing regions' message. If no
+ namespace or table is specified, it will verify all existing regions.
+ It accepts a combination of multiple namespace and tables. Table names
+ should include the namespace portion, even for tables in the default
+ namespace, otherwise it will assume as a namespace value.
+ An example triggering missing regions report for tables 'table_1'
+ and 'table_2', under default namespace:
+ $ HBCK2 reportMissingRegionsInMeta default:table_1 default:table_2
+ An example triggering missing regions report for table 'table_1'
+ under default namespace, and for all tables from namespace 'ns1':
+ $ HBCK2 reportMissingRegionsInMeta default:table_1 ns1
+ Returns list of missing regions for each table passed as parameter, or
+ for each table on namespaces specified as parameter.
+
setRegionState <ENCODED_REGIONNAME> <STATE>
Possible region states:
OFFLINE, OPENING, OPEN, CLOSING, CLOSED, SPLITTING, SPLIT,
@@ -202,6 +253,10 @@ _HBCK2_ versions should be able to work across multiple
hbase-2 releases. It wil
fail with a complaint if it is unable to run. There is no `HbckService` in
versions
of hbase before 2.0.3 and 2.1.1. _HBCK2_ will not work against these versions.
+As _HBCK2_ evolves independently from _HBase_ main project, there will be
eventually the need to
+define new fix methods with client side implementations (at least until a
related one can be added
+on Master's `HbckService` facade), so that _HBCK2_ can operate on such _HBase_
releases without
+requiring a cluster upgrade. One example of such methods is the
_setRegionState_.
## Finding Problems
@@ -419,9 +474,60 @@ To schedule an assign for the hbase:namespace table noted
in the above log line,
```HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase
org.apache.hbase.HBCK2 -skip assigns 9559cf72b8e81e1291c626a8e781a6ae```
... passing the encoded name for the namespace region (the encoded name will
differ per deploy).
-### hbase:meta region/table restore/rebuild
+### Missing Regions in META - hbase:meta region/table restore/rebuild
+
+There's been some extra-ordinary cases where table regions are removed from
META table.
+Some triage on such cases revealed those were operator-induced, after execution
+attempts of the obsolete *hbck1* _OfflineMetaRepair_ tool. _OfflineMetaRepair_
is a well known tool
+for fixing META table related issues on HBase 1.x versions. The original
version is not compatible
+with HBase 2.x or higher versions, and it has undergone some adjustments to be
now run within hbck2.
+
+In most of these cases, regions may be missing in meta at random, but hbase
may still be
+operational. In such situations, problem can be addressed with master online,
using _addFsRegionsMissingInMeta_ command.
+This command is less disruptive to hbase than the full meta rebuild covered
later, and it can be used even for
+recovering _namespace_ table region.
+
+#### Online meta rebuild recipe
+
+If meta corruption is not too critical, hbase would still be able to bring it
online. Even if namespace region
+is among the missing ones in meta, it will still be possible to scan meta in
the initialization period,
+where master will be waiting for namespace to be assigned. To verify on this,
a meta scan command can be executed
+as below. If it does not time out or show any errors, _meta_ is online:
+
+```
+echo "scan 'hbase:meta', {COLUMN=>'info:regioninfo'}" | hbase shell
+```
+
+HBCK2 _addFsRegionsMissingInMeta_ can be used if the above does not show any
errors. It reads region
+metadata info available on the FS region dirs, in order to re-create regions
in META. Since it can
+run with hbase partially operational, it attempts to disable online tables
that are affected by the
+reported problem and is gonna have regions re-added to _meta_.
+It can check for specific tables/namespaces, or all tables
+from all namespaces. An example adding missing regions for tables 'tbl_1' on
default namespace,
+'tbl_2' on namespace 'n1' and for all tables from namespace 'n2':
+
+```
+$ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2
+```
+
+As it operates independently from Master, once it finishes successfully,
additional steps are
+required to actually have the re-added regions assigned. These are listed
below:
+
+1. _addFsRegionsMissingInMeta_ outputs an _assigns_ command with all regions
that got re-added. This
+command needs to be executed later, so copy and save it for convenience.
+
+2. For HBase versions prior to 2.3.0, after _addFsRegionsMissingInMeta_
finished successfully and output has been saved,
+restart all running HBase Masters.
+
+3. Once Master's are restarted and META is already online (check if Web UI is
accessible), run
+_assigns_ command from _addFsRegionsMissingInMeta_ output saved per
instructions from #1.
+
+NOTE: If _namespace_ region is among the missing ones, you will need to add
_--skip_ flag at the
+beginning of _assigns_ command returned.
+
-Should a cluster suffer a catastrophic loss of the `hbase:meta` region, a
rough rebuild is possible following the below recipe. In outline: stop the
cluster; run the _OfflineMetaRepair_ tool which reads directories and metadata
dropped into the filesystem making a best effort at reconstructing a viable
_hbase:meta_ table; restart your cluster; inject an assign to bring the system
namespace table online; and then finally, re-assign userspace tables you'd like
enabled (the rebuilt _hbase:m [...]
+Should a cluster suffer a catastrophic loss of the `hbase:meta` region, a
rough rebuild is possible following the below recipe.
+In outline: stop the cluster; run the _OfflineMetaRepair_ tool which reads
directories and metadata dropped into the filesystem making a best effort at
reconstructing a viable _hbase:meta_ table; restart your cluster; inject an
assign to bring the system namespace table online; and then finally, re-assign
userspace tables you'd like enabled (the rebuilt _hbase:meta_ creates a table
with all tables offline and no regions assigned).
#### Detailed rebuild recipe
Stop the cluster.
diff --git a/hbase-hbck2/pom.xml b/hbase-hbck2/pom.xml
index 5de2b11..1f61653 100644
--- a/hbase-hbck2/pom.xml
+++ b/hbase-hbck2/pom.xml
@@ -214,6 +214,12 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <version>2.1.0</version>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<profiles>
diff --git
a/hbase-hbck2/src/main/java/org/apache/hbase/FsRegionsMetaRecoverer.java
b/hbase-hbck2/src/main/java/org/apache/hbase/FsRegionsMetaRecoverer.java
new file mode 100644
index 0000000..977ca77
--- /dev/null
+++ b/hbase-hbck2/src/main/java/org/apache/hbase/FsRegionsMetaRecoverer.java
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
+import org.apache.hadoop.hbase.util.FSUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * This class implements the inner works required for check and recover
regions that wrongly
+ * went missing in META.
+ * Normally HBCK2 fix options rely on Master self-contained information to
recover/fix
+ * inconsistencies, but this an exception case where META table is in a broken
state.
+ * So, it assumes HDFS state as the source of truth, in other words, methods
provided here consider
+ * meta information found on HDFS region dirs as the valid ones.
+ */
+public class FsRegionsMetaRecoverer implements Closeable {
+ private static final Logger LOG =
LogManager.getLogger(FsRegionsMetaRecoverer.class);
+ private final FileSystem fs;
+ private final Connection conn;
+ private final Configuration config;
+
+ public FsRegionsMetaRecoverer(Configuration configuration) throws
IOException {
+ this.config = configuration;
+ this.fs = CommonFSUtils.getRootDirFileSystem(configuration);
+ this.conn = ConnectionFactory.createConnection(configuration);
+ }
+
+ /*Initially defined for test only purposes */
+ FsRegionsMetaRecoverer(Configuration configuration, Connection connection,
FileSystem fileSystem){
+ this.config = configuration;
+ this.conn = connection;
+ this.fs = fileSystem;
+ }
+
+ private List<Path> getTableRegionsDirs(String table) throws Exception {
+ String hbaseRoot = this.config.get(HConstants.HBASE_DIR);
+ Path tableDir = FSUtils.getTableDir(new Path(hbaseRoot),
TableName.valueOf(table));
+ return FSUtils.getRegionDirs(fs, tableDir);
+ }
+
+ public Map<TableName,List<Path>> reportTablesMissingRegions(final
List<String> namespacesOrTables)
+ throws IOException {
+ final Map<TableName,List<Path>> result = new HashMap<>();
+ List<TableName> tableNames =
MetaTableAccessor.getTableStates(this.conn).keySet().stream()
+ .filter(tableName -> {
+ if(namespacesOrTables==null || namespacesOrTables.isEmpty()){
+ return true;
+ } else {
+ Optional<String> findings = namespacesOrTables.stream().filter(
+ name -> (name.indexOf(":") > 0) ?
+ tableName.equals(TableName.valueOf(name)) :
+ tableName.getNamespaceAsString().equals(name)).findFirst();
+ return findings.isPresent();
+ }
+ }).collect(Collectors.toList());
+ tableNames.stream().forEach(tableName -> {
+ try {
+ result.put(tableName,
+
findMissingRegionsInMETA(tableName.getNameWithNamespaceInclAsString()));
+ } catch (Exception e) {
+ LOG.warn(e);
+ }
+ });
+ return result;
+ }
+
+ List<Path> findMissingRegionsInMETA(String table) throws Exception {
+ final List<Path> missingRegions = new ArrayList<>();
+ final List<Path> regionsDirs = getTableRegionsDirs(table);
+ TableName tableName = TableName.valueOf(table);
+ List<RegionInfo> regionInfos = MetaTableAccessor.
+ getTableRegions(this.conn, tableName, false);
+ HashSet<String> regionsInMeta = regionInfos.stream().map(info ->
+ info.getEncodedName()).collect(Collectors.toCollection(HashSet::new));
+ for(final Path regionDir : regionsDirs){
+ if (!regionsInMeta.contains(regionDir.getName())) {
+ LOG.debug(regionDir + "is not in META.");
+ missingRegions.add(regionDir);
+ }
+ }
+ return missingRegions;
+ }
+
+ public void putRegionInfoFromHdfsInMeta(Path region) throws IOException {
+ RegionInfo info = HRegionFileSystem.loadRegionInfoFileContent(fs, region);
+ MetaTableAccessor.addRegionToMeta(conn, info);
+ }
+
+ @Override public void close() throws IOException {
+ this.conn.close();
+ }
+}
diff --git a/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
b/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
index 3a4bd9a..0a83622 100644
--- a/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
+++ b/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java
@@ -25,11 +25,18 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.List;
+import java.util.Map;
import java.util.Properties;
import java.util.stream.Collectors;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.ClusterMetrics;
import org.apache.hadoop.hbase.CompareOperator;
import org.apache.hadoop.hbase.HBaseConfiguration;
@@ -48,6 +55,7 @@ import org.apache.hadoop.hbase.client.TableState;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.filter.SubstringComparator;
import org.apache.hadoop.hbase.master.RegionState;
+import org.apache.hadoop.hbase.util.Pair;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
@@ -90,6 +98,10 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
private static final String SCHEDULE_RECOVERIES = "scheduleRecoveries";
private static final String FIX_META = "fixMeta";
+ private static final String ADD_MISSING_REGIONS_IN_META_FOR_TABLES =
+ "addFsRegionsMissingInMeta";
+ private static final String ADD_MISSING_REGIONS_IN_META =
"addMissingRegionsInMeta";
+ private static final String REPORT_MISSING_REGIONS_IN_META =
"reportMissingRegionsInMeta";
private Configuration conf;
static String [] MINIMUM_HBCK2_VERSION = {"2.0.3", "2.1.1", "2.2.0",
"3.0.0"};
private boolean skipCheck = false;
@@ -163,6 +175,87 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
return EXIT_FAILURE;
}
+ Map<TableName,List<Path>> reportTablesWithMissingRegionsInMeta(String...
nameSpaceOrTable)
+ throws IOException {
+ Map<TableName,List<Path>> report;
+ try(final FsRegionsMetaRecoverer fsRegionsMetaRecoverer = new
FsRegionsMetaRecoverer(this.conf)){
+ List<String> names = nameSpaceOrTable != null ?
Arrays.asList(nameSpaceOrTable) : null;
+ report = fsRegionsMetaRecoverer.reportTablesMissingRegions(names);
+ } catch (IOException e) {
+ LOG.error("Error reporting missing regions: ", e);
+ throw e;
+ }
+ if(LOG.isDebugEnabled()) {
+ LOG.debug(formatMissingRegionsInMetaReport(report));
+ }
+ return report;
+ }
+
+ List<String> addMissingRegionsInMeta(List<Path> regionsPath) throws
IOException {
+ List<String> reAddedRegionsEncodedNames = new ArrayList<>();
+ try(final FsRegionsMetaRecoverer fsRegionsMetaRecoverer = new
FsRegionsMetaRecoverer(this.conf)){
+ for(Path regionPath : regionsPath){
+ fsRegionsMetaRecoverer.putRegionInfoFromHdfsInMeta(regionPath);
+ reAddedRegionsEncodedNames.add(regionPath.getName());
+ }
+ }
+ return reAddedRegionsEncodedNames;
+ }
+
+ Pair<List<String>, List<ExecutionException>>
addMissingRegionsInMetaForTables(String...
+ nameSpaceOrTable) throws IOException {
+ ExecutorService executorService = Executors.newFixedThreadPool(
+ (nameSpaceOrTable == null ||
+ nameSpaceOrTable.length > Runtime.getRuntime().availableProcessors()) ?
+ Runtime.getRuntime().availableProcessors() :
+ nameSpaceOrTable.length);
+ List<Future<List<String>>> futures = new ArrayList<>( nameSpaceOrTable ==
null ? 1 :
+ nameSpaceOrTable.length);
+ final List<String> readdedRegionNames = new ArrayList<>();
+ List<ExecutionException> executionErrors = new ArrayList<>();
+ try {
+ //reducing number of retries in case disable fails due to namespace
table region also missing
+ this.conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);
+ try(ClusterConnection conn = connect();
+ final Admin admin = conn.getAdmin()) {
+ Map<TableName,List<Path>> report =
reportTablesWithMissingRegionsInMeta(nameSpaceOrTable);
+ for (TableName tableName : report.keySet()) {
+ if(admin.tableExists(tableName)) {
+ futures.add(executorService.submit(new Callable<List<String>>() {
+ @Override
+ public List<String> call() throws Exception {
+ LOG.debug("running thread for {}",
tableName.getNameWithNamespaceInclAsString());
+ return addMissingRegionsInMeta(report.get(tableName));
+ }
+ }));
+ } else {
+ LOG.warn("Table {} does not exist! Skipping...",
+ tableName.getNameWithNamespaceInclAsString());
+ }
+ }
+ for(Future<List<String>> f : futures){
+ try {
+ readdedRegionNames.addAll(f.get());
+ } catch (ExecutionException e){
+ //we want to allow potential running threads to finish, so we
collect execution
+ //errors and show those later
+ LOG.debug("Caught execution error: ", e);
+ executionErrors.add(e);
+ }
+ }
+ }
+ } catch (IOException | InterruptedException e) {
+ LOG.error("ERROR executing thread: ", e);
+ throw new IOException(e);
+ } finally {
+ executorService.shutdown();
+ }
+ Pair<List<String>, List<ExecutionException>> result = new Pair<>();
+ result.setFirst(readdedRegionNames);
+ result.setSecond(executionErrors);
+ return result;
+ }
+
List<Long> assigns(Hbck hbck, String [] args) throws IOException {
Options options = new Options();
Option override = Option.builder("o").longOpt("override").build();
@@ -267,6 +360,34 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
StringWriter sw = new StringWriter();
PrintWriter writer = new PrintWriter(sw);
writer.println("Command:");
+ writer.println(" " + ADD_MISSING_REGIONS_IN_META_FOR_TABLES + "
<NAMESPACE|"
+ + "NAMESPACE:TABLENAME>...");
+ writer.println(" Options:");
+ writer.println(" -d,--force_disable aborts fix for table if disable
fails.");
+ writer.println(" To be used in scenarios where some regions may be
missing in META,");
+ writer.println(" but there's still a valid 'regioninfo' metadata file on
HDFS. ");
+ writer.println(" This is a lighter version of 'OfflineMetaRepair tool
commonly used for ");
+ writer.println(" similar issues on 1.x release line. ");
+ writer.println(" This command needs META to be online. For each table
name passed as");
+ writer.println(" parameter, it performs a diff between regions available
in META, ");
+ writer.println(" against existing regions dirs on HDFS. Then, for region
dirs with ");
+ writer.println(" no matches in META, it reads regioninfo metadata file
and ");
+ writer.println(" re-creates given region in META. Regions are re-created
in 'CLOSED' ");
+ writer.println(" state at META table only, but not in Masters' cache,
and are not ");
+ writer.println(" assigned either. To get these regions online, run HBCK2
'assigns'command ");
+ writer.println(" printed at the end of this command results for
convenience.");
+ writer.println();
+ writer.println(" NOTE: If using hbase releases older than 2.3.0, a
rolling restart of ");
+ writer.println(" HMasters is needed prior to executing the provided
'assigns' command. ");
+ writer.println();
+ writer.println(" An example adding missing regions for tables 'tbl_1' on
default ");
+ writer.println(" namespace, 'tbl_2' on namespace 'n1' and for all tables
from ");
+ writer.println(" namespace 'n2': ");
+ writer.println(" $ HBCK2 " + ADD_MISSING_REGIONS_IN_META_FOR_TABLES +
+ " default:tbl_1 n1:tbl_2 n2 ");
+ writer.println(" Returns HBCK2 'assigns' command with all re-inserted
regions.");
+ writer.println(" SEE ALSO: " + REPORT_MISSING_REGIONS_IN_META);
+ writer.println();
writer.println(" " + ASSIGNS + " [OPTIONS] <ENCODED_REGIONNAME>...");
writer.println(" Options:");
writer.println(" -o,--override override ownership by another
procedure");
@@ -318,6 +439,33 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
writer.println(" '--fix' option. Pass a table name to check for
replication barrier and");
writer.println(" purge if '--fix'.");
writer.println();
+ writer.println(" " + REPORT_MISSING_REGIONS_IN_META + " <NAMESPACE|"
+ + "NAMESPACE:TABLENAME>...");
+ writer.println(" To be used in scenarios where some regions may be
missing in META,");
+ writer.println(" but there's still a valid 'regioninfo metadata file on
HDFS. ");
+ writer.println(" This is a checking only method, designed for reporting
purposes and");
+ writer.println(" doesn't perform any fixes, providing a view of which
regions (if any) ");
+ writer.println(" would get re-added to meta, grouped by respective
table/namespace. ");
+ writer.println(" To effectively re-add regions in meta, "
+ + ADD_MISSING_REGIONS_IN_META_FOR_TABLES + " should be executed. ");
+ writer.println(" This command needs META to be online. For each
namespace/table passed");
+ writer.println(" as parameter, it performs a diff between regions
available in META, ");
+ writer.println(" against existing regions dirs on HDFS. Region dirs with
no matches");
+ writer.println(" are printed grouped under its related table name.
Tables with no");
+ writer.println(" missing regions will show a 'no missing regions'
message. If no");
+ writer.println(" namespace or table is specified, it will verify all
existing regions.");
+ writer.println(" It accepts a combination of multiple namespace and
tables. Table names");
+ writer.println(" should include the namespace portion, even for tables
in the default");
+ writer.println(" namespace, otherwise it will assume as a namespace
value.");
+ writer.println(" An example triggering missing regions report for tables
'table_1'");
+ writer.println(" and 'table_2', under default namespace:");
+ writer.println(" $ HBCK2 reportMissingRegionsInMeta default:table_1
default:table_2");
+ writer.println(" An example triggering missing regions report for table
'table_1'");
+ writer.println(" under default namespace, and for all tables from
namespace 'ns1':");
+ writer.println(" $ HBCK2 reportMissingRegionsInMeta default:table_1
ns1");
+ writer.println(" Returns list of missing regions for each table passed
as parameter, or ");
+ writer.println(" for each table on namespaces specified as parameter.");
+ writer.println();
writer.println(" " + SET_REGION_STATE + " <ENCODED_REGIONNAME> <STATE>");
writer.println(" Possible region states:");
writer.println(" OFFLINE, OPENING, OPEN, CLOSING, CLOSED, SPLITTING,
SPLIT,");
@@ -369,6 +517,7 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
writer.println();
writer.println(" SEE ALSO, org.apache.hbase.hbck1.OfflineMetaRepair, the
offline");
writer.println(" hbase:meta tool. See the HBCK2 README for how to use.");
+ writer.println();
writer.close();
return sw.toString();
}
@@ -604,6 +753,26 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
}
break;
+ case ADD_MISSING_REGIONS_IN_META_FOR_TABLES:
+ if(commands.length < 2){
+ showErrorMessage(command + " takes one or more table names.");
+ return EXIT_FAILURE;
+ }
+ Pair<List<String>, List<ExecutionException>> result =
+ addMissingRegionsInMetaForTables(purgeFirst(commands));
+
System.out.println(formatReAddedRegionsMessage(result.getFirst(),result.getSecond()));
+ break;
+
+ case REPORT_MISSING_REGIONS_IN_META:
+ try {
+ Map<TableName,List<Path>> report =
+ reportTablesWithMissingRegionsInMeta(purgeFirst(commands));
+ System.out.println(formatMissingRegionsInMetaReport(report));
+ } catch (Exception e) {
+ return EXIT_FAILURE;
+ }
+ break;
+
default:
showErrorMessage("Unsupported command: " + command);
return EXIT_FAILURE;
@@ -615,6 +784,51 @@ public class HBCK2 extends Configured implements
org.apache.hadoop.util.Tool {
return things.stream().map(Object::toString).collect(Collectors.joining(",
"));
}
+ private String formatMissingRegionsInMetaReport(Map<TableName,List<Path>>
report) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append("Missing Regions for each table:\n\t");
+ report.keySet().stream().forEach(table -> {
+ builder.append(table);
+ if (!report.get(table).isEmpty()){
+ builder.append("->\n\t\t");
+ report.get(table).stream().forEach(region ->
builder.append(region.getName())
+ .append(" "));
+ } else {
+ builder.append(" -> No missing regions");
+ }
+ builder.append("\n\t");
+ });
+ return builder.toString();
+ }
+
+ private String formatReAddedRegionsMessage(List<String> readdedRegionNames,
+ List<ExecutionException> executionErrors) {
+ final StringBuilder finalText = new StringBuilder();
+ finalText.append("Regions re-added into Meta:
").append(readdedRegionNames.size());
+ if(!readdedRegionNames.isEmpty()){
+ finalText.append("\n")
+ .append("WARNING: \n\t")
+ .append(readdedRegionNames.size()).append(" regions were added ")
+ .append("to META, but these are not yet on Masters cache. \n")
+ .append("You need to restart Masters, then run hbck2 'assigns' command
below:\n\t\t")
+ .append(buildHbck2AssignsCommand(readdedRegionNames));
+ }
+ if(!executionErrors.isEmpty()){
+ finalText.append("\n")
+ .append("ERROR: \n\t")
+ .append("There were following errors on at least one table thread:\n");
+ executionErrors.stream().forEach(e ->
finalText.append(e.getMessage()).append("\n"));
+ }
+ return finalText.toString();
+ }
+
+ private String buildHbck2AssignsCommand(List<String> regions) {
+ final StringBuilder builder = new StringBuilder();
+ builder.append("assigns ");
+ regions.forEach(region -> builder.append(region).append(" "));
+ return builder.toString();
+ }
+
/**
* @return A new array with first element dropped.
*/
diff --git
a/hbase-hbck2/src/test/java/org/apache/hbase/TestFsRegionsMetaRecoverer.java
b/hbase-hbck2/src/test/java/org/apache/hbase/TestFsRegionsMetaRecoverer.java
new file mode 100644
index 0000000..c06e92d
--- /dev/null
+++ b/hbase-hbck2/src/test/java/org/apache/hbase/TestFsRegionsMetaRecoverer.java
@@ -0,0 +1,209 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hbase;
+
+import static org.junit.Assert.assertEquals;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSInputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.RegionInfo;
+import org.apache.hadoop.hbase.client.RegionInfoBuilder;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.ResultScanner;
+import org.apache.hadoop.hbase.client.Scan;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableState;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.HBaseProtos;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hbase.FsRegionsMetaRecoverer;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class TestFsRegionsMetaRecoverer {
+
+ private Connection mockedConnection;
+ private FileSystem mockedFileSystem;
+ private Table mockedTable;
+ private FsRegionsMetaRecoverer fixer;
+ private String testTblDir;
+
+ @Before
+ public void setup() throws Exception {
+ this.mockedConnection = Mockito.mock(Connection.class);
+ this.mockedFileSystem = Mockito.mock(FileSystem.class);
+ this.mockedTable = Mockito.mock(Table.class);
+ Configuration config = HBaseConfiguration.create();
+ Mockito.when(this.mockedConnection.getConfiguration()).thenReturn(config);
+
Mockito.when(this.mockedConnection.getTable(TableName.META_TABLE_NAME)).thenReturn(mockedTable);
+ this.testTblDir = config.get(HConstants.HBASE_DIR) +
"/data/default/test-tbl";
+ this.fixer = new FsRegionsMetaRecoverer(config, mockedConnection,
mockedFileSystem);
+ }
+
+ private RegionInfo createRegionInfo(String table){
+ long regionTS = System.currentTimeMillis();
+ RegionInfo info = RegionInfoBuilder.newBuilder(TableName.valueOf(table))
+ .setRegionId(regionTS)
+ .build();
+ return info;
+ }
+
+ private Cell createCellForRegionInfo(RegionInfo info){
+ byte[] regionInfoValue =
ProtobufUtil.prependPBMagic(ProtobufUtil.toRegionInfo(info)
+ .toByteArray());
+ Cell cell = CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY)
+ .setRow(info.getRegionName())
+ .setFamily(Bytes.toBytes("info"))
+ .setQualifier(Bytes.toBytes("regioninfo"))
+ .setType(Cell.Type.Put)
+ .setValue(regionInfoValue)
+ .build();
+ return cell;
+ }
+
+ private Cell createCellForTableState(TableName tableName){
+ Cell cell = CellBuilderFactory.create(CellBuilderType.SHALLOW_COPY)
+ .setRow(tableName.getName())
+ .setFamily(Bytes.toBytes("table"))
+ .setQualifier(Bytes.toBytes("state"))
+ .setType(Cell.Type.Put)
+ .setValue(HBaseProtos.TableState.newBuilder()
+ .setState(TableState.State.ENABLED.convert()).build().toByteArray())
+ .build();
+ return cell;
+ }
+
+ @Test
+ public void testFindMissingRegionsInMETANoMissing() throws Exception {
+ ResultScanner mockedRS = Mockito.mock(ResultScanner.class);
+
Mockito.when(this.mockedTable.getScanner(Mockito.any(Scan.class))).thenReturn(mockedRS);
+ RegionInfo info = createRegionInfo("test-tbl");
+ List<Cell> cells = new ArrayList();
+ cells.add(createCellForRegionInfo(info));
+ Result result = Result.create(cells);
+ Mockito.when(mockedRS.next()).thenReturn(result,null);
+ FileStatus status = new FileStatus();
+ status.setPath(new Path(this.testTblDir + "/" + info.getEncodedName()));
+ Mockito.when(mockedFileSystem.listStatus(new Path(this.testTblDir)))
+ .thenReturn(new FileStatus[]{status});
+ assertEquals("Should had returned 0 missing regions",
+ 0, fixer.findMissingRegionsInMETA("test-tbl").size());
+ }
+
+ @Test
+ public void testFindMissingRegionsInMETAOneMissing() throws Exception {
+ ResultScanner mockedRS = Mockito.mock(ResultScanner.class);
+
Mockito.when(this.mockedTable.getScanner(Mockito.any(Scan.class))).thenReturn(mockedRS);
+ List<Cell> cells = new ArrayList();
+ Result result = Result.create(cells);
+ Mockito.when(mockedRS.next()).thenReturn(result,null);
+ Path p = new Path(this.testTblDir+ "/182182182121");
+ FileStatus status = new FileStatus(0, true, 0, 0,0, p);
+ Mockito.when(mockedFileSystem.listStatus(new Path(this.testTblDir)))
+ .thenReturn(new FileStatus[]{status});
+ List<Path> missingRegions = fixer.findMissingRegionsInMETA("test-tbl");
+ assertEquals("Should had returned 1 missing region",
+ 1, missingRegions.size());
+ assertEquals(p,missingRegions.get(0));
+ }
+
+ @Test
+ public void testPutRegionInfoFromHdfsInMeta() throws Exception {
+ RegionInfo info = this.createRegionInfo("test-tbl");
+ Path regionPath = new Path("/hbase/data/default/test-tbl/" +
info.getEncodedName());
+ FSDataInputStream fis = new FSDataInputStream(new
TestInputStreamSeekable(info));
+ Mockito.when(this.mockedFileSystem.open(new Path(regionPath,
".regioninfo")))
+ .thenReturn(fis);
+ fixer.putRegionInfoFromHdfsInMeta(regionPath);
+ Mockito.verify(this.mockedConnection).getTable(TableName.META_TABLE_NAME);
+ Mockito.verify(this.mockedTable).put(Mockito.any(Put.class));
+ }
+
+ @Test
+ public void testReportTablesMissingRegionsOneMissing() throws Exception {
+ ResultScanner mockedRS = Mockito.mock(ResultScanner.class);
+
Mockito.when(this.mockedTable.getScanner(Mockito.any(Scan.class))).thenReturn(mockedRS);
+ List<Cell> cells = new ArrayList();
+ cells.add(createCellForTableState(TableName.valueOf("test-tbl")));
+ Result result = Result.create(cells);
+ Mockito.when(mockedRS.next()).thenReturn(result,null);
+ FileStatus status = new FileStatus();
+ Path p = new Path(this.testTblDir+ "/182182182121");
+ status.setPath(p);
+ Mockito.when(mockedFileSystem.listStatus(new Path(this.testTblDir)))
+ .thenReturn(new FileStatus[]{status});
+ Mockito.when(this.mockedConnection.getTable(TableName.META_TABLE_NAME))
+ .thenReturn(this.mockedTable);
+ Map<TableName, List<Path>> report = fixer.reportTablesMissingRegions(null);
+ assertEquals("Should had returned 1 missing region",
+ 1,report.size());
+ }
+
+ private class TestInputStreamSeekable extends FSInputStream {
+
+ private ByteArrayInputStream in;
+ private long length;
+
+ private TestInputStreamSeekable(RegionInfo info) throws Exception {
+ byte[] bytes = RegionInfo.toDelimitedByteArray(info);
+ this.length = bytes.length;
+ this.in = new ByteArrayInputStream(bytes);
+ }
+
+ @Override
+ public void seek(long l) throws IOException {
+ this.in.skip(l);
+ }
+
+ @Override
+ public long getPos() throws IOException {
+ return this.length - in.available();
+ }
+
+ @Override
+ public boolean seekToNewSource(long l) throws IOException {
+ this.in.skip(l);
+ return true;
+ }
+
+ @Override
+ public int read() throws IOException {
+ return in.read();
+ }
+ }
+
+}
+
diff --git a/hbase-hbck2/src/test/java/org/apache/hbase/TestHBCK2.java
b/hbase-hbck2/src/test/java/org/apache/hbase/TestHBCK2.java
index 1516503..0c423c3 100644
--- a/hbase-hbck2/src/test/java/org/apache/hbase/TestHBCK2.java
+++ b/hbase-hbck2/src/test/java/org/apache/hbase/TestHBCK2.java
@@ -25,11 +25,14 @@ import java.util.List;
import java.util.stream.Collectors;
import junit.framework.TestCase;
+import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.MetaTableAccessor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.ClusterConnection;
+import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Hbck;
import org.apache.hadoop.hbase.client.RegionInfo;
@@ -41,9 +44,19 @@ import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.logging.log4j.LogManager;
import org.junit.AfterClass;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
+import org.junit.Rule;
import org.junit.Test;
+import org.junit.rules.TestName;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.util.Map;
+
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
/**
* Tests commands. For command-line parsing, see adjacent test.
@@ -56,6 +69,9 @@ public class TestHBCK2 {
private static final TableName REGION_STATES_TABLE_NAME = TableName.
valueOf(TestHBCK2.class.getSimpleName() + "-REGIONS_STATES");
+ @Rule
+ public TestName testName = new TestName();
+
/**
* A 'connected' hbck2 instance.
*/
@@ -175,6 +191,128 @@ public class TestHBCK2 {
}
}
+ @Test
+ public void testAddMissingRegionsInMetaAllRegionsMissing() throws Exception {
+ this.testAddMissingRegionsInMetaForTables(5,5);
+ }
+
+ @Test
+ public void testAddMissingRegionsInMetaTwoMissingOnly() throws Exception {
+ this.testAddMissingRegionsInMetaForTables(2,5);
+ }
+
+ @Test
+ public void testReportMissingRegionsInMetaAllNsTbls() throws Exception {
+ this.testReportMissingRegionsInMeta(5, 5,null);
+ }
+
+ @Test
+ public void testReportMissingRegionsInMetaSpecificTbl() throws Exception {
+ this.testReportMissingRegionsInMeta(5, 5,
+ TABLE_NAME.getNameWithNamespaceInclAsString());
+ }
+
+ @Test
+ public void testReportMissingRegionsInMetaSpecificTblAndNsTbl() throws
Exception {
+ this.testReportMissingRegionsInMeta(5, 5,
+ TABLE_NAME.getNameWithNamespaceInclAsString(), "hbase:namespace");
+ }
+
+ @Test
+ public void testReportMissingRegionsInMetaSpecificTblAndNsTblAlsoMissing()
throws Exception {
+ List<RegionInfo> regions = MetaTableAccessor
+ .getTableRegions(TEST_UTIL.getConnection(),
TableName.valueOf("hbase:namespace"));
+ HBCKMetaTableAccessor.deleteRegionInfo(TEST_UTIL.getConnection(),
regions.get(0));
+ this.testReportMissingRegionsInMeta(5, 6,
+ TABLE_NAME.getNameWithNamespaceInclAsString(), "hbase:namespace");
+ }
+
+ @Test
+ public void testFormatReportMissingRegionsInMetaNoMissing() throws
IOException {
+ final String expectedResult = "Missing Regions for each table:\n"
+ + "\tTestHBCK2 -> No missing regions\n\thbase:namespace -> No missing
regions\n\t\n";
+ String result = testFormatMissingRegionsInMetaReport();
+ assertTrue(result.contains(expectedResult));
+ }
+
+ @Test
+ public void testFormatReportMissingInMetaOneMissing() throws IOException {
+ TableName tableName = createTestTable(5);
+ List<RegionInfo> regions = MetaTableAccessor
+ .getTableRegions(TEST_UTIL.getConnection(), tableName);
+ HBCKMetaTableAccessor.deleteRegionInfo(TEST_UTIL.getConnection(),
regions.get(0));
+ String expectedResult = "Missing Regions for each table:\n";
+ String result = testFormatMissingRegionsInMetaReport();
+ //validates initial report message
+ assertTrue(result.contains(expectedResult));
+ //validates our test table region is reported missing
+ expectedResult = "\t" + tableName.getNameAsString() + "->\n\t\t"
+ + regions.get(0).getEncodedName();
+ assertTrue(result.contains(expectedResult));
+ //validates namespace region is not reported missing
+ expectedResult = "\n\thbase:namespace -> No missing regions\n\t";
+ assertTrue(result.contains(expectedResult));
+ }
+
+ private String testFormatMissingRegionsInMetaReport()
+ throws IOException {
+ HBCK2 hbck = new HBCK2(TEST_UTIL.getConfiguration());
+ final StringBuilder builder = new StringBuilder();
+ PrintStream originalOS = System.out;
+ OutputStream testOS = new OutputStream() {
+ @Override public void write(int b) throws IOException {
+ builder.append((char)b);
+ }
+ };
+ System.setOut(new PrintStream(testOS));
+
+ hbck.run(new String[]{"reportMissingRegionsInMeta"});
+ System.setOut(originalOS);
+ return builder.toString();
+ }
+
+ private TableName createTestTable(int totalRegions) throws IOException {
+ TableName tableName = TableName.valueOf(testName.getMethodName());
+ TEST_UTIL.createMultiRegionTable(tableName, Bytes.toBytes("family1"),
totalRegions);
+ return tableName;
+ }
+
+ private void testAddMissingRegionsInMetaForTables(int missingRegions, int
totalRegions)
+ throws Exception {
+ TableName tableName = createTestTable(totalRegions);
+ HBCK2 hbck = new HBCK2(TEST_UTIL.getConfiguration());
+ List<RegionInfo> regions = MetaTableAccessor
+ .getTableRegions(TEST_UTIL.getConnection(), tableName);
+ Connection connection = TEST_UTIL.getConnection();
+ regions.subList(0, missingRegions).forEach( r ->
deleteRegionInfo(connection, r));
+ int remaining = totalRegions - missingRegions;
+ assertEquals("Table should have " + remaining + " regions in META.",
remaining,
+ MetaTableAccessor.getRegionCount(TEST_UTIL.getConnection(), tableName));
+
assertEquals(missingRegions,hbck.addMissingRegionsInMetaForTables("default:"
+ + tableName.getNameAsString()).getFirst().size());
+ assertEquals("Table regions should had been re-added in META.",
totalRegions,
+ MetaTableAccessor.getRegionCount(TEST_UTIL.getConnection(), tableName));
+ //compare the added regions to make sure those are the same
+ List<RegionInfo> newRegions = MetaTableAccessor
+ .getTableRegions(TEST_UTIL.getConnection(), tableName);
+ assertEquals("All re-added regions should be the same", regions,
newRegions);
+ }
+
+ private void testReportMissingRegionsInMeta(int missingRegionsInTestTbl,
+ int expectedTotalMissingRegions, String... namespaceOrTable) throws
Exception {
+ List<RegionInfo> regions = MetaTableAccessor
+ .getTableRegions(TEST_UTIL.getConnection(), TABLE_NAME);
+ Connection connection = TEST_UTIL.getConnection();
+ regions.subList(0, missingRegionsInTestTbl).forEach( r ->
deleteRegionInfo(connection, r));
+ HBCK2 hbck = new HBCK2(TEST_UTIL.getConfiguration());
+ final Map<TableName,List<Path>> report =
+ hbck.reportTablesWithMissingRegionsInMeta(namespaceOrTable);
+ long resultingMissingRegions = report.keySet().stream().mapToLong( nsTbl ->
+ report.get(nsTbl).size()).sum();
+ assertEquals(expectedTotalMissingRegions, resultingMissingRegions);
+ hbck.addMissingRegionsInMetaForTables(null);
+ }
+
@Test (expected = IllegalArgumentException.class)
public void testSetRegionStateInvalidRegionAndInvalidState() throws
IOException {
try (ClusterConnection connection = this.hbck2.connect()) {
@@ -202,4 +340,13 @@ public class TestHBCK2 {
}
}
}
+
+ private void deleteRegionInfo(Connection connection, RegionInfo region) {
+ try {
+ HBCKMetaTableAccessor.deleteRegionInfo(connection, region);
+ } catch (IOException e) {
+ fail(e.getMessage());
+ }
+ }
+
}
