I have uploaded a patch in HBASE-9346 to add some check into HBCK. Working pretty well.
To solve the issue, I "simply" merged the culprit region with the next one, so there is no more issues with the region boundaries. Now I will major compact and check the result. Then I will take a look at the NPE below... 2013/8/26 Jean-Marc Spaggiari <[email protected]> > I have created > HBASE-9346<https://issues.apache.org/jira/browse/HBASE-9346>for the > boundaries issues. > > > 2013/8/26 Jean-Marc Spaggiari <[email protected]> > >> You're right. I figured that after. So I removed the oldlogs (might be a >> left over, was there for 2 days even after multiple stop/restart) but I'm >> still getting this error on the region: >> >> >> Exception in thread "main" java.lang.NullPointerException >> at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716) >> at >> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138) >> at >> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149) >> at >> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325) >> at >> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241) >> at >> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196) >> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756) >> >> Even if now I'm able to get the keys information: >> firstKey=\x16\xCE\x8E\xCF\x00\x00\x00\x00... >> lastKey=\x19\xA0Ma\x00\x00\x00\x00... >> >> This is another region that the one where I have overlaps.... >> >> JM >> >> >> 2013/8/26 Sergey Shelukhin <[email protected]> >> >>> Um, I may be missing something here, but is it really an HFile? >>> Looks like an hlog to me. >>> >>> org.apache.hadoop.hbase.io. >>> hfile.CorruptHFileException: Problem reading >>> HFile Trailer from file >>> >>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526 >>> >>> >>> On Mon, Aug 26, 2013 at 12:31 PM, Jean-Marc Spaggiari < >>> [email protected]> wrote: >>> >>> > Hi, >>> > >>> > Don't ask me how, but I have one table in a pretty strange state. >>> > >>> > First, seems that I have one corrupted HFile (at least). >>> > >>> > FirstKey return null. (Same for StopKey since header is corrupted). >>> > >>> > Exception in thread "main" java.lang.NullPointerException >>> > at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:325) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:241) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196) >>> > at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756) >>> > >>> > Because the header is not correct. >>> > >>> > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading >>> > HFile Trailer from file >>> > >>> > >>> hdfs://node3:9000/hbase/work_proposed/db83e64f34a5a608335818321f1a6c32/.oldlogs/hlog.1377344531526 >>> > at >>> > >>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:595) >>> > at >>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:642) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196) >>> > at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:756) >>> > Caused by: java.lang.IllegalArgumentException: Invalid HFile version: >>> > 16275367 (expected to be between 1 and 2) >>> > at >>> > >>> org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:771) >>> > at >>> > >>> > >>> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323) >>> > at >>> > >>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:549) >>> > ... 5 more >>> > >>> > HBCK don't detect that: >>> > 0 inconsistencies detected. >>> > Status: OK >>> > >>> > Also, I have an issue with the first key / last key reported by the >>> stores >>> > overlapping when the one reported by the META are not. >>> > >>> > Store: >>> > firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00... >>> > lastKey=\xFF\xFF\xFF\xFE\x00\x00\x00\x00... >>> > Meta: >>> > firstKey=\xF5\x9A\xEA&\x00\x00\x00\x00 >>> > lastKey=\xF5\x9B@}\x00\x00\x00\x00... >>> > >>> > >>> > So, few things. >>> > >>> > 1) We should add something into HBCK to check the HFile format again >>> > corruption. >>> > 2) We should add something into HBCK to validate META regions >>> boundaries >>> > against Store Files >>> > 3) How can I repaire my HFile? ;) >>> > >>> > I'm already working on #2 and will have something ready soon. Then I >>> will >>> > most probably more to #1. But I only have detection dont for now. I'm >>> not >>> > sure exactly what are the correct steps to repair... >>> > >>> > JM >>> > >>> >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified >>> that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender >>> immediately >>> and delete it from your system. Thank You. >>> >> >> >
