[
https://issues.apache.org/jira/browse/HBASE-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201441#comment-17201441
]
Martin Braun commented on HBASE-22976:
--------------------------------------
Is there a workaround for reading in WAL files from recovered.edits? How can I
achieve this?
I have an issue with hbase 2.2.5 (and hadoop-2.8.5) after a full disk event I
have 38 inconsistencies, when I do a
hbase --internal-classpath hbck
I get a bunch of these errors:
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
tt_ix_bizStep_inserting in hdfs dir
hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8a1acb499bf454b072daeee5960daa73!
It may be an invalid format or version file. Treating as an orphaned regiondir.
ERROR: Orphan region in HDFS: Unable to load .regioninfo from table
tt_ix_bizStep_inserting in hdfs dir
hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8f64025b68958ebddeb812297facdfc6!
It may be an invalid format or version file. Treating as an orphaned regiondir.
When looking into these directories I see that there is indeed no .regioninfo
file:
hdfs dfs -ls -R
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0
drwxr-xr-x - jenkins supergroup 0 2020-09-21 11:23
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits
-rw-r--r-- 3 jenkins supergroup 74133 2020-09-21 11:11
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/0000000000000000285
-rw-r--r-- 3 jenkins supergroup 74413 2020-09-16 19:03
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/0000000000000000286
-rw-r--r-- 3 jenkins supergroup 74693 2020-09-16 19:05
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/0000000000000000287
-rw-r--r-- 3 jenkins supergroup 79427 2020-09-16 18:27
hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/0000000000000000305
So I have now a bunch of recovered.edits WAL files I would like to replay - but
how?
The WALPlayer is not able to replay recovered.edits files, the source code
http://hbase.apache.org/2.2/devapidocs/src-html/org/apache/hadoop/hbase/mapreduce/WALInputFormat.html
seems to expect an endtime coded into the filename:
long fileStartTime = Long.parseLong(name.substring(idx+1));
323 if (fileStartTime <= endTime) {
324 LOG.info("Found: " + file);
325 result.add(file);
326 }
327 } catch (NumberFormatException x) {
328 idx = 0;
But the files in recovered.edits are named differently (just a numbers like
00000000000000195).
Would renaming of the files help? But with which endtime?
> [HBCK2] Add RecoveredEditsPlayer
> --------------------------------
>
> Key: HBASE-22976
> URL: https://issues.apache.org/jira/browse/HBASE-22976
> Project: HBase
> Issue Type: Sub-task
> Components: hbck2
> Reporter: Michael Stack
> Priority: Major
>
> We need a recovered edits player. Messing w/ the 'adoption service' --
> tooling to adopt orphan regions and hfiles -- I've been manufacturing damaged
> clusters by moving stuff around under the running cluster. No reason to think
> that an hbase couldn't lose accounting of a whole region if a cataclysm. If
> so, region will have stuff like the '.regioninfo', dirs per column family w/
> store files but it could too have a 'recovered_edits' directory with content
> in it. We have a WALPlayer for errant WALs. We have the FSHLog tool which can
> read recovered_edits content for debugging data loss. Missing is a
> RecoveredEditsPlayer.
> I took a look at extending the WALPlayer since it has a bunch of nice options
> and it can run at bulk. Ideally, it would just digest recovered edits content
> if passed an option or recovered edits directories. On first glance, it
> didn't seem like an easy integration.... Would be worth taking a look again.
> Would be good if we could avoid making a new, distinct tool, just for
> Recovered Edits.
> The bulkload tool expects hfiles in column family directories. Recovered
> edits files are not hfiles and the files are x-columnfamily so this is not
> the way to go though a bulkload-like tool that moved the recovered edits
> files under the appropriate region dir and asked the region reopen would be a
> possibility (Would need the bulk load complete trick of splitting input if
> the region boundaries in the live cluster do not align w/ those of the errant
> recovered edits files).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
