[jira] [Commented] (HBASE-24920) A tool to rewrite corrupted HFiles
[
https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188273#comment-17188273
]
Wellington Chevreuil commented on HBASE-24920:
--
{quotes}completely re-implement needed pieces in hbase operator tool so that
there are no dependencies on hbase version (making this tool hard to maintain).
{quotes}
We've been doing this already in some parts of operator tools code (take a look
at HBCKMetaTableAccessor and HBCKFsUtils, which are the operator tools dups of
hbase MetaTableAccessor and FsUtils). If the efforts of porting/reimplement
these private classes is not too complex, I would suggest to go with this
option.
> A tool to rewrite corrupted HFiles
> --
>
> Key: HBASE-24920
> URL: https://issues.apache.org/jira/browse/HBASE-24920
> Project: HBase
> Issue Type: Brainstorming
> Components: hbase-operator-tools
>Reporter: Andrey Elenskiy
>Priority: Major
>
> Typically I have been dealing with corrupted HFiles (due to loss of hdfs
> blocks) by just removing them. However, It always seemed wasteful to throw
> away the entire HFile (which can be hundreds of gigabytes), just because one
> hdfs block is missing (128MB).
> I think there's a possibility for a tool that can rewrite an HFile by
> skipping corrupted blocks.
> There can be multiple types of issues with hdfs blocks but any of them can be
> treated as if the block doesn't exist:
> 1. All the replicas can be lost
> 2. The block can be corrupted due to some bug in hdfs (I've recently run into
> HDFS-15186 by experimenting with EC).
> At the simplest the tool can be a local mapreduce job (mapper only) with a
> custom HFile reader input that can seek to next DATABLK to skip corrupted
> hdfs blocks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24920) A tool to rewrite corrupted HFiles
[ https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188033#comment-17188033 ] Andrey Elenskiy commented on HBASE-24920: - Was away for a week and just attempted to write a prototype. The tool is fairly simple: if HFileScanner.next() throws exception, search for DATABLK magic offset in FSInputStream and resume from there. However, I realized that it wouldn't be possible to have it be part of hbase operator tools. I rely on some private classes in org.apache.hadoop.hbase.io.hfile package which change quite a bit between versions (2.2.5, 2.3.0, master are all different) making it impossible as a drop in jar. So, there are couple of choices here: 1. package the tool with dependencies and pin to some hbase version (which one?) 2. package along with hbase being part of "hbase hfile" command (probably why [~stack] suggested it in the first place ;)) 3. completely re-implement needed pieces in hbase operator tool so that there are no dependencies on hbase version (making this tool hard to maintain). > A tool to rewrite corrupted HFiles > -- > > Key: HBASE-24920 > URL: https://issues.apache.org/jira/browse/HBASE-24920 > Project: HBase > Issue Type: Brainstorming > Components: hbase-operator-tools >Reporter: Andrey Elenskiy >Priority: Major > > Typically I have been dealing with corrupted HFiles (due to loss of hdfs > blocks) by just removing them. However, It always seemed wasteful to throw > away the entire HFile (which can be hundreds of gigabytes), just because one > hdfs block is missing (128MB). > I think there's a possibility for a tool that can rewrite an HFile by > skipping corrupted blocks. > There can be multiple types of issues with hdfs blocks but any of them can be > treated as if the block doesn't exist: > 1. All the replicas can be lost > 2. The block can be corrupted due to some bug in hdfs (I've recently run into > HDFS-15186 by experimenting with EC). > At the simplest the tool can be a local mapreduce job (mapper only) with a > custom HFile reader input that can seek to next DATABLK to skip corrupted > hdfs blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24920) A tool to rewrite corrupted HFiles
[ https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181487#comment-17181487 ] Michael Stack commented on HBASE-24920: --- Sounds like a good plan to me [~timoha] > A tool to rewrite corrupted HFiles > -- > > Key: HBASE-24920 > URL: https://issues.apache.org/jira/browse/HBASE-24920 > Project: HBase > Issue Type: Brainstorming > Components: hbase-operator-tools >Reporter: Andrey Elenskiy >Priority: Major > > Typically I have been dealing with corrupted HFiles (due to loss of hdfs > blocks) by just removing them. However, It always seemed wasteful to throw > away the entire HFile (which can be hundreds of gigabytes), just because one > hdfs block is missing (128MB). > I think there's a possibility for a tool that can rewrite an HFile by > skipping corrupted blocks. > There can be multiple types of issues with hdfs blocks but any of them can be > treated as if the block doesn't exist: > 1. All the replicas can be lost > 2. The block can be corrupted due to some bug in hdfs (I've recently run into > HDFS-15186 by experimenting with EC). > At the simplest the tool can be a local mapreduce job (mapper only) with a > custom HFile reader input that can seek to next DATABLK to skip corrupted > hdfs blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24920) A tool to rewrite corrupted HFiles
[ https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181463#comment-17181463 ] Andrey Elenskiy commented on HBASE-24920: - I currently like that `hbase hfile` is read only and allows to debug things (I use it for debugging the corrupt blocks at the moment), the name of the class is even HFilePrettyPrinter. On the mailing list Sean suggested to put it in the repo with `hbase-operator-tools` and I prefer that as it aligns better with the goal of that repo (correct bugs/inconsistencies via operator intervention). What do you think? > A tool to rewrite corrupted HFiles > -- > > Key: HBASE-24920 > URL: https://issues.apache.org/jira/browse/HBASE-24920 > Project: HBase > Issue Type: Brainstorming > Components: hbase-operator-tools >Reporter: Andrey Elenskiy >Priority: Major > > Typically I have been dealing with corrupted HFiles (due to loss of hdfs > blocks) by just removing them. However, It always seemed wasteful to throw > away the entire HFile (which can be hundreds of gigabytes), just because one > hdfs block is missing (128MB). > I think there's a possibility for a tool that can rewrite an HFile by > skipping corrupted blocks. > There can be multiple types of issues with hdfs blocks but any of them can be > treated as if the block doesn't exist: > 1. All the replicas can be lost > 2. The block can be corrupted due to some bug in hdfs (I've recently run into > HDFS-15186 by experimenting with EC). > At the simplest the tool can be a local mapreduce job (mapper only) with a > custom HFile reader input that can seek to next DATABLK to skip corrupted > hdfs blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24920) A tool to rewrite corrupted HFiles
[ https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181449#comment-17181449 ] Michael Stack commented on HBASE-24920: --- Do you think it would be an option on [http://hbase.apache.org/book.html#hfile_tool] [~timoha] ? > A tool to rewrite corrupted HFiles > -- > > Key: HBASE-24920 > URL: https://issues.apache.org/jira/browse/HBASE-24920 > Project: HBase > Issue Type: Brainstorming > Components: hbase-operator-tools >Reporter: Andrey Elenskiy >Priority: Major > > Typically I have been dealing with corrupted HFiles (due to loss of hdfs > blocks) by just removing them. However, It always seemed wasteful to throw > away the entire HFile (which can be hundreds of gigabytes), just because one > hdfs block is missing (128MB). > I think there's a possibility for a tool that can rewrite an HFile by > skipping corrupted blocks. > There can be multiple types of issues with hdfs blocks but any of them can be > treated as if the block doesn't exist: > 1. All the replicas can be lost > 2. The block can be corrupted due to some bug in hdfs (I've recently run into > HDFS-15186 by experimenting with EC). > At the simplest the tool can be a local mapreduce job (mapper only) with a > custom HFile reader input that can seek to next DATABLK to skip corrupted > hdfs blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
