[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13465506#comment-13465506 ] Fenng Wang commented on HBASE-6871: --- [~stack],the new patch can really fix my issue, my test script has successfully imported the hfiles in the attachment without any exception. HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.92.3, 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 6871-0.94.txt, 6871-hfile-index-0.92.txt, 6871-hfile-index-0.92-v2.txt, 6871.txt, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1,
[jira] [Commented] (HBASE-6871) HFileBlockIndex Write Error BlockIndex in HFile V2
[ https://issues.apache.org/jira/browse/HBASE-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464533#comment-13464533 ] Fenng Wang commented on HBASE-6871: --- [~mikhail]：Your explanation is helpful for me, my patch can only solve this particular issue, the difference between root size and nonroot size of BlockIndexChunk maybe trigger this bug again, I will use the new patch to resolve my issue completely, thank you! HFileBlockIndex Write Error BlockIndex in HFile V2 -- Key: HBASE-6871 URL: https://issues.apache.org/jira/browse/HBASE-6871 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.94.1 Environment: redhat 5u4 Reporter: Fenng Wang Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 428a400628ae412ca45d39fce15241fd.hfile, 6871.txt, 787179746cc347ce9bb36f1989d17419.hfile, 960a026ca370464f84903ea58114bc75.hfile, d0026fa8d59b4df291718f59dd145aad.hfile, D5703.1.patch, D5703.2.patch, D5703.3.patch, D5703.4.patch, D5703.5.patch, hbase-6871-0.94.patch, ImportHFile.java, test_hfile_block_index.sh After writing some data, compaction and scan operation both failure, the exception message is below: 2012-09-18 06:32:26,227 ERROR org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: Compaction failed regionName=hfile_test,,1347778722498.d220df43fb9d8af4633bd7f547613f9e., storeName=page_info, fileCount=7, fileSize=1.3m (188.0k, 188.0k, 188.0k, 188.0k, 188.0k, 185.8k, 223.3k), priority=9, time=45826250816757428java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for reader reader=hdfs://hadoopdev1.cm6:9000/hbase/hfile_test/d220df43fb9d8af4633bd7f547613f9e/page_info/b0f6118f58de47ad9d87cac438ee0895, compression=lzo, cacheConf=CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false], firstKey=http://com.truereligionbrandjeans.www/Womens_Dresses/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Shirts/pl/c/Womens_Shirts/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/Womens_Sweaters/pl/c/4010.html/page_info:anchor_sig/1347764439449/DeleteColumn, lastKey=http://com.trura.www//page_info:page_type/1347763395089/Put, avgKeyLen=776, avgValueLen=4, entries=12853, length=228611, cur=http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/1347764003865/Put/vlen=1/ts=0] to key http://com.truereligionbrandjeans.www/Womens_Exclusive_Details/pl/c/4970.html/page_info:is_deleted/OLDEST_TIMESTAMP/Minimum/vlen=0/ts=0 at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:178) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402) at org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1570) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:997) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1216) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:250) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Expected block type LEAF_INDEX, but got INTERMEDIATE_INDEX: blockType=INTERMEDIATE_INDEX, onDiskSizeWithoutHeader=8514, uncompressedSizeWithoutHeader=131837, prevBlockOffset=-1,
