Hi,
We met a weird scenario in our production environment.
IndexOutOfBoundsException is thrown while retrieving mid key of the storefile
after region compaction.
Log Snippet :
-----------------
2016-05-30 01:41:58,484 | INFO |
regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | Completed
compaction of 1 (all) file(s) in CF of
User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for store is
22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to execute.
|
org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HStore.java:1356)
2016-05-30 01:41:58,485 | INFO |
regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | Completed
compaction: Request =
regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, time=295643974900644;
duration=6mins, 25sec |
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:544)
2016-05-30 01:41:58,529 | ERROR |
regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | Compaction
failed Request =
regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, time=295643974900644 |
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:563)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:540)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
at
org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
at
org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
at
org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-----------------
Observation:
>> HFilePrettyPrinter also print the message "Unable to retrieve the midkey"
>> for the mid key.
>> HDFS fsck report the hfile healthy.
Though successful region compaction were also there, only few region compaction
failed with same error.
Have anyone faced this issue? Any help will be appreciated.
HBase version is 1.0.0.
Regards,
Pankaj
