[
https://issues.apache.org/jira/browse/HBASE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795090#action_12795090
]
Greg Lu commented on HBASE-2073:
--------------------------------
Another thing that might be worth mentioning is that the regionserver died
overnight, so it's possible it happened during a major compaction.
> IllegalArgumentException causing regionserver failure
> -----------------------------------------------------
>
> Key: HBASE-2073
> URL: https://issues.apache.org/jira/browse/HBASE-2073
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.2
> Environment: Ubuntu 8.10, Java 1.6.0_10, HBase 0.20.2
> Reporter: Greg Lu
> Attachments: hbase-hadoop-regionserver-factory05.lab.mtl.log
>
>
> After a regionserver went down last night, I checked its logs and found the
> following exception:
> 2009-12-29 00:17:27,663 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
> /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262060247637,
> entries=1830, calcsize=22946017, filesize=22758899. New hlog
> /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262063847659
> 2009-12-29 00:34:36,210 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:218)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-12-29 00:34:36,214 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020, call next(4170645244799815171, 1) from
> 192.168.1.108:53401: error: java.io.IOException:
> java.lang.IllegalArgumentException
> java.io.IOException: java.lang.IllegalArgumentException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:218)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> ... 5 more
> Looks like this bug was encountered before at
> https://issues.apache.org/jira/browse/HBASE-1495 and spanned a few JIRAs.
> It's supposed to be resolved as of 0.20.0, but we're running 0.20.2 and it
> took down one of our regionservers.
> I'm also attaching more of the log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
