Thanks Ted for replying. Yeah, We have a plan to upgrade. But currently I want to know the reason behind this. I tried to reproduce this in our test environment but didn’t happen.
in HFilePrettyPrinter output "numDataIndexLevels=2", so there were multilevel data index. Is which circumstances this problem can happen? Regards, Pankaj -----Original Message----- From: Ted Yu [mailto:[email protected]] Sent: Saturday, June 04, 2016 12:16 AM To: [email protected] Cc: bhupendra jain; Sharanabasappa G Keriwaddi Subject: Re: IndexOutOfBoundsException during retrieving region split point 1.0.0 is quite old. Is it possible to upgrade to 1.1 or 1.2 release ? Thanks On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr <[email protected]> wrote: > Hi, > > We met a weird scenario in our production environment. > IndexOutOfBoundsException is thrown while retrieving mid key of the > storefile after region compaction. > > Log Snippet : > ----------------- > 2016-05-30 01:41:58,484 | INFO | > regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | > Completed compaction of 1 (all) file(s) in CF of > User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5. > into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for > store is > 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to > execute. | > org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HS > tore.java:1356) > 2016-05-30 01:41:58,485 | INFO | > regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | > Completed compaction: Request = > regionName=User_Namespace:User_Table,100050007010803_20140126_30801071 > 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3 > c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, > time=295643974900644; duration=6mins, 25sec | > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn > er.run(CompactSplitThread.java:544) > 2016-05-30 01:41:58,529 | ERROR | > regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | > Compaction failed Request = > regionName=User_Namespace:User_Table,100050007010803_20140126_30801071 > 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3 > c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, > time=295643974900644 | > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn > er.run(CompactSplitThread.java:563) > java.lang.IndexOutOfBoundsException > at java.nio.Buffer.checkIndex(Buffer.java:540) > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) > at > org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490) > at > org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480) > at > org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685) > at > org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126) > at > org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986) > at > org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > ----------------- > Observation: > >> HFilePrettyPrinter also print the message "Unable to retrieve the > midkey" for the mid key. > >> HDFS fsck report the hfile healthy. > > Though successful region compaction were also there, only few region > compaction failed with same error. > > Have anyone faced this issue? Any help will be appreciated. > HBase version is 1.0.0. > > Regards, > Pankaj >
