[
https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464265#comment-13464265
]
Phabricator commented on HBASE-6871:
------------------------------------
lhofhansl has commented on the revision "[jira] [HBASE-6871] [89-fb] Block
index corruption test case and fix".
Can't pretend to fully understands what going on in the test, but if it
reproduces the problem that's perfect. Thanks for doing this Mikhail.
The fix is different from the one proposed on the issue.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java:988 Are
these expected exceptional cases?
If not, should we have asserts instead?
REVISION DETAIL
https://reviews.facebook.net/D5703
BRANCH
repro_interm_index_bug_v7
To: lhofhansl, Kannan, Liyin, stack, JIRA, mbautin
> HFileBlockIndex Write Error BlockIndex in HFile V2
> --------------------------------------------------
>
> Key: HBASE-6871
> URL: https://issues.apache.org/jira/browse/HBASE-6871
> Project: HBase
> Issue Type: Bug
> Components: HFile
> Affects Versions: 0.94.1
> Environment: redhat 5u4
> Reporter: Fenng Wang
> Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: 428a400628ae412ca45d39fce15241fd.hfile,
> 787179746cc347ce9bb36f1989d17419.hfile,
> 960a026ca370464f84903ea58114bc75.hfile,
> d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch,
> D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch,
> ImportHFile.java, test_hfile_block_index.sh
>
>
> After writing some data, compaction and scan operation both failure, the
> exception message is below:
> 2012-09-18 06:32:26,227 ERROR
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
> Compaction failed
> regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e.,
> storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k,
> 188.0k, 188.0k, 185.8k, 223.3k), priority=9,
> time=45826250816757428java.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895,
> compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true]
> [cacheDataOnWrite=false] [cacheIndexesOnWrite=false]
> [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false],
> firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn,
> lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put,
> avgKeyLen=776, avgValueLen=4, entries=12853, length=228611,
> cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0]
> to key
> http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178)
>
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402)
> at
> org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570)
>
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216)
> at
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got
> INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX,
> onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837,
> prevBlockOffset=-1,
> dataBeginsWith=\x00\x00\x00\x9B\x00\x00\x00\x00\x00\x00\x03#\x00\x00\x050\x00\x00\x08\xB7\x00\x00\x0Cr\x00\x00\x0F\xFA\x00\x00\x120,
> fileOffset=218942 at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.validateBlockType(HFileReaderV2.java:378)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:331)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
> After some debug works，I found that when hfile closing, if the rootChunk is
> empty, the only one curInlineChunk will upgrade to root chunk. But if the
> last block flushing make curInlineChunk exceed max index block size, the root
> chunk(upgrade from curInlineChunk) will be splited into intermediate index
> blocks, and the index level is set to 2. So when BlockIndexReader read the
> root index, it expects the next level index block is leaf index(index
> level=2), but the on disk index block is intermediate block, the error
> happened.
> After I add some code to check curInlineChunk's size when rootChunk is empty
> in shouldWriteBlock(boolean closing), this bug can be fixed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
