saintstack commented on a change in pull request #6: HBASE-22680 [HBCK2]
OfflineMetaRepair for hbase2/hbck2
URL: https://github.com/apache/hbase-operator-tools/pull/6#discussion_r303963317
##########
File path: hbase-hbck2/README.md
##########
@@ -337,11 +334,53 @@ The Master is unable to continue startup because there
is no Procedure to assign
_hbase:meta_ (or _hbase:namespace_). To inject one, use the _HBCK2_ tool:
```
-HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase
org.apache.hbase.HBCK2 assigns 1588230740
+HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase
org.apache.hbase.HBCK2 assigns -skip 1588230740
```
-...where 1588230740 is the encoded name of the _hbase:meta_ Region.
+...where 1588230740 is the encoded name of the _hbase:meta_ Region. Pass the
'-skip' option to
+stop HBCK2 doing a verstion check against the remote master. If the remote
master is not up,
+the version check will prompt a 'Master is initializing response' or
'PleaseHoldException'
+and drop the assign attempt. The '-skip' command punts on version check and
will land the
+scheduled assign.
The same may happen to the _hbase:namespace_ system table. Look for the
encoded Region name of the _hbase:namespace_ Region and do similar to
-what we did for _hbase:meta_.
+what we did for _hbase:meta_. In this latter case, the Master actually
+prints out a helpful message that looks like the following:
+
+```2019-07-09 22:08:38,966 WARN [master/localhost:16000:becomeActiveMaster]
master.HMaster:
hbase:namespace,,1562733904278.9559cf72b8e81e1291c626a8e781a6ae. is NOT online;
state={9559cf72b8e81e1291c626a8e781a6ae state=CLOSED, ts=1562735318897,
server=null}; ServerCrashProcedures=true. Master startup cannot progress, in
holding-pattern until region onlined.```
+
+To schedule an assign for the hbase:namespace table noted in the above log
line, you would do:
+```HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase
org.apache.hbase.HBCK2 -skip assigns 9559cf72b8e81e1291c626a8e781a6ae```
+... passing the encoded name for the namespace region (the encoded name will
differ per deploy).
+
+### hbase:meta region/table restore/rebuild
+
+Should a cluster suffer a catastrophic loss of the `hbase:meta` region, a
rough rebuild is possible following the below receipe. In outline: stop the
cluster; run the _OfflineMetaRepair_ tool which reads directories and metadata
dropped into the filesystem making a best effort at reconstructing a viable
_hbase:meta_ table; restart your cluster; inject an assign to bring the system
namespace table online; and then finally, re-assign userspace tables you'd like
enabled (the rebuilt _hbase:meta_ creates a table with all tables offline and
no regions assigned).
Review comment:
Good one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
