[ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497029#comment-15497029
 ] 

Mikhail Antonov commented on HBASE-16644:
-----------------------------------------

[~stack] thanks for quick reply! So far my thoughts on it:

 - that seems to only happen under specific conditions, since I've only seen a 
handful of such cases. Still looking at what's special about those files. So 
far it *seems" that it only happens on specific blocks, the code path in 
HFileReaderV2 constructor when we read meta index (data index was read fine).
 - Debugging and reproducing is kind of painful as I've only seen that on real 
files and don't yet have cooked-up test file that can demonstrate this effect 
on.
 - Regardless of how such a file was written, still if we can read some file on 
1.2 and we can't on 1.3 that is a critical regression on the read path IMO.

> Errors when reading legit HFile' Trailer on branch 1.3
> ------------------------------------------------------
>
>                 Key: HBASE-16644
>                 URL: https://issues.apache.org/jira/browse/HBASE-16644
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Mikhail Antonov
>            Assignee: Mikhail Antonov
>            Priority: Critical
>             Fix For: 1.3.0
>
>         Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  <file name>
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>       at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1164)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>       at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>       at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>       at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>       at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>       ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>       at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>       at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.<init>(HFileBlock.java:344)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:156)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file <file path>
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>       at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1164)
>       at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.<init>(HalfStoreFileReader.java:104)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>       at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>       at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>       at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>       at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>       ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
> read 1072
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:156)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> In my case this problem was reproducible by running `hbase hfile -m  -f` 
> command. There seem to be two changes in behavior introduced by HBASE-15477 
> (cc [~stack]).
> One is that HFileBlock#getOnDiskSizeWithHeader always assumes that checksum 
> verification is true (this behavior results in onDiskSizeWithHeader being 
> calculated differently in 1.2 and 1.3 for some cases).
> Second is that in HFileBlock constructor we always attempt to retrive 
> checksum-related fields from the header even if no checksum verification is 
> going on.
> Attached is the patch which when applied allows 1.3 to read again the same 
> HFiles.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to