In ur test env also u have numDataIndexLevels=2? Or it is 1 only? -Anoop-
On Mon, Jun 6, 2016 at 1:12 PM, Pankaj kr <[email protected]> wrote: > Thanks Ted for replying. > Yeah, We have a plan to upgrade. But currently I want to know the reason > behind this. I tried to reproduce this in our test environment but didn’t > happen. > > in HFilePrettyPrinter output "numDataIndexLevels=2", so there were multilevel > data index. Is which circumstances this problem can happen? > > Regards, > Pankaj > > -----Original Message----- > From: Ted Yu [mailto:[email protected]] > Sent: Saturday, June 04, 2016 12:16 AM > To: [email protected] > Cc: bhupendra jain; Sharanabasappa G Keriwaddi > Subject: Re: IndexOutOfBoundsException during retrieving region split point > > 1.0.0 is quite old. > > Is it possible to upgrade to 1.1 or 1.2 release ? > > Thanks > > On Fri, Jun 3, 2016 at 8:12 AM, Pankaj kr <[email protected]> wrote: > >> Hi, >> >> We met a weird scenario in our production environment. >> IndexOutOfBoundsException is thrown while retrieving mid key of the >> storefile after region compaction. >> >> Log Snippet : >> ----------------- >> 2016-05-30 01:41:58,484 | INFO | >> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | >> Completed compaction of 1 (all) file(s) in CF of >> User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5. >> into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for >> store is >> 22.0 G. This selection was in queue for 0sec, and took 6mins, 25sec to >> execute. | >> org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HS >> tore.java:1356) >> 2016-05-30 01:41:58,485 | INFO | >> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | >> Completed compaction: Request = >> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071 >> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3 >> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, >> time=295643974900644; duration=6mins, 25sec | >> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn >> er.run(CompactSplitThread.java:544) >> 2016-05-30 01:41:58,529 | ERROR | >> regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749 | >> Compaction failed Request = >> regionName=User_Namespace:User_Table,100050007010803_20140126_30801071 >> 7550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3 >> c5., storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, >> time=295643974900644 | >> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunn >> er.run(CompactSplitThread.java:563) >> java.lang.IndexOutOfBoundsException >> at java.nio.Buffer.checkIndex(Buffer.java:540) >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139) >> at >> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490) >> at >> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349) >> at >> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512) >> at >> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480) >> at >> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685) >> at >> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126) >> at >> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986) >> at >> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82) >> at >> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914) >> at >> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240) >> at >> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> ----------------- >> Observation: >> >> HFilePrettyPrinter also print the message "Unable to retrieve the >> midkey" for the mid key. >> >> HDFS fsck report the hfile healthy. >> >> Though successful region compaction were also there, only few region >> compaction failed with same error. >> >> Have anyone faced this issue? Any help will be appreciated. >> HBase version is 1.0.0. >> >> Regards, >> Pankaj >>
