You're right. I figured that after. So I removed the oldlogs (might be a
left over, was there for 2 days even after multiple stop/restart) but I'm
still getting this error on the region:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
at
org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
Even if now I'm able to get the keys information:
firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00...
lastKey=\x19\xA0Ma\x00\x00\x00\x00...
This is another region that the one where I have overlaps....
JM
2013/8/26 Sergey Shelukhin <[email protected]>
> Um, I may be missing something here, but is it really an HFile?
> Looks like an hlog to me.
>
> org.apache.hadoop.hbase.io.
> hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
>
> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
>
>
> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari <
> [email protected]> wrote:
>
> > Hi,
> >
> > Don't ask me how, but I have one table in a pretty strange state.
> >
> > First, seems that I have one corrupted HFile (at least).
> >
> > FirstKey return null. (Same for StopKey since header is corrupted).
> >
> > Exception in thread "main" java.lang.NullPointerException
> > at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
> > at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
> >
> > Because the header is not correct.
> >
> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> > HFile Trailer from file
> >
> >
> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526
> > at
> > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595)
> > at
> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
> > at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756)
> > Caused by: java.lang.IllegalArgumentException: Invalid HFile version:
> > 16275367 (expected to be between 1 and 2)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549)
> > ... 5 more
> >
> > HBCK don't detect that:
> > 0 inconsistencies detected.
> > Status: OK
> >
> > Also, I have an issue with the first key / last key reported by the
> stores
> > overlapping when the one reported by the META are not.
> >
> > Store:
> > firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00...
> > lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00...
> > Meta:
> > firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00
> > lastKey=\xF5\x9B@}\x00\x00\x00\x00...
> >
> >
> > So, few things.
> >
> > 1) We should add something into HBCK to check the HFile format again
> > corruption.
> > 2) We should add something into HBCK to validate META regions boundaries
> > against Store Files
> > 3) How can I repaire my HFile? ;)
> >
> > I'm already working on #2 and will have something ready soon. Then I will
> > most probably more to #1. But I only have detection dont for now. I'm not
> > sure exactly what are the correct steps to repair...
> >
> > JM
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
