[
https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464056#comment-13464056
]
Mikhail Bautin commented on HBASE-6871:
---------------------------------------
[~feng wang]: nice catch! My understanding of this bug is somewhat different
from yours, though. There are no new data blocks being written after
shouldWriteBlock(true) is called when closing an HFileWriterV2. What I think is
happening is that a new block gets written and a new entry is added to the root
index right before writeInlineBlocks(true) is called in HFileWriterV2.close. If
the "closing" parameter was set to false, a new leaf-level block would have
been written at this point. However, as you described, the current index chunk
gets promoted to a root index and then split into intermediate index blocks,
which is incorrect, because in this case there are no leaf index blocks.
I think your patch fixes this particular issue, but another issue remains. We
use different formats for non-root and root index blocks, and the non-root
index format is slightly more space-efficient. Thus, it is still possible that
the current index chunk is promoted to become the root index chunk because its
"non-root" size is under the configured index chunk size, but its "root" size
turns out to be above the configured index chunk size, so we create
intermediate blocks in writeIndexBlocks and run into the same problem.
I think the correct solution is that once we decide to promote the index chunk
to the root chunk, we should never split it into intermediate blocks. That will
fix both issues. I will try to come up with a test and a patch today.
> HFileBlockIndex Write Error BlockIndex in HFile V2
> --------------------------------------------------
>
> Key: HBASE-6871
> URL: https://issues.apache.org/jira/browse/HBASE-6871
> Project: HBase
> Issue Type: Bug
> Components: HFile
> Affects Versions: 0.94.1
> Environment: redhat 5u4
> Reporter: Fenng Wang
> Priority: Critical
> Attachments: 428a400628ae412ca45d39fce15241fd.hfile,
> 787179746cc347ce9bb36f1989d17419.hfile,
> 960a026ca370464f84903ea58114bc75.hfile,
> d0026fa8d59b4df291718f59dd145aad.hfile, hbase-6871-0.94.patch,
> ImportHFile.java, test_hfile_block_index.sh
>
>
> After writing some data, compaction and scan operation both failure, the
> exception message is below:
> 2012-09-18 06:32:26,227 ERROR
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> Compaction failed
> regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e.,
> storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k,
> 188.0k, 188.0k, 185.8k, 223.3k), priority=9,
> time=45826250816757428java.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895,
> compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
> firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn,
> lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put,
> avgKeyLen=776, avgValueLen=4, entries=12853, length=228611,
> cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0]
> to key
> http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178)
>
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402)
> at
> org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570)
>
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216)
> at
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got
> INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX,
> onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837,
> prevBlockOffset=-1,
> dataBeginsWith=\x00\x00\x00\x9B\x00\x00\x00\x00\x00\x00\x03#\x00\x00\x050\x00\x00\x08\xB7\x00\x00\x0Cr\x00\x00\x0F\xFA\x00\x00\x120,
> fileOffset=218942 at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.validateBlockType(HFileReaderV2.java:378)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:331)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
> After some debug works，I found that when hfile closing, if the rootChunk is
> empty, the only one curInlineChunk will upgrade to root chunk. But if the
> last block flushing make curInlineChunk exceed max index block size, the root
> chunk(upgrade from curInlineChunk) will be splited into intermediate index
> blocks, and the index level is set to 2. So when BlockIndexReader read the
> root index, it expects the next level index block is leaf index(index
> level=2), but the on disk index block is intermediate block, the error
> happened.
> After I add some code to check curInlineChunk's size when rootChunk is empty
> in shouldWriteBlock(boolean closing), this bug can be fixed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
