IllegalArgumentException causing regionserver failure
-----------------------------------------------------
Key: HBASE-2073
URL: https://issues.apache.org/jira/browse/HBASE-2073
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.20.2
Environment: Ubuntu 8.10, Java 1.6.0_10, HBase 0.20.2
Reporter: Greg Lu
Attachments: hbase-hadoop-regionserver-factory05.lab.mtl.log
After a regionserver went down last night, I checked its logs and found the
following exception:
2009-12-29 00:17:27,663 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
/hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262060247637,
entries=1830, calcsize=22946017, filesize=22758899. New hlog
/hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262063847659
2009-12-29 00:34:36,210 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2009-12-29 00:34:36,214 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60020, call next(4170645244799815171, 1) from 192.168.1.108:53401:
error: java.io.IOException: java.lang.IllegalArgumentException
java.io.IOException: java.lang.IllegalArgumentException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
... 5 more
Looks like this bug was encountered before at
https://issues.apache.org/jira/browse/HBASE-1495 and spanned a few JIRAs. It's
supposed to be resolved as of 0.20.0, but we're running 0.20.2 and it took down
one of our regionservers.
I'm also attaching more of the log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
