[
https://issues.apache.org/jira/browse/HBASE-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796424#action_12796424
]
ryan rawson commented on HBASE-2073:
------------------------------------
the log doesnt have much detail, typically in the past these kinds of errors
can indicate short read from HDFS. I'm not seeing any of those in this logfile,
is there anything interesting in the HDFS logs?
For this error to have happened, there should have been a short read in the
block, which is "impossible".
> IllegalArgumentException causing regionserver failure
> -----------------------------------------------------
>
> Key: HBASE-2073
> URL: https://issues.apache.org/jira/browse/HBASE-2073
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.2
> Environment: Ubuntu 8.10, Java 1.6.0_10, HBase 0.20.2
> Reporter: Greg Lu
> Assignee: ryan rawson
> Fix For: 0.20.3
>
> Attachments: hbase-hadoop-regionserver-factory05.lab.mtl.log
>
>
> After a regionserver went down last night, I checked its logs and found the
> following exception:
> 2009-12-29 00:17:27,663 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll
> /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262060247637,
> entries=1830, calcsize=22946017, filesize=22758899. New hlog
> /hbase/amsterdam_factory/.logs/factory05.lab.mtl,60020,1262042255724/hlog.dat.1262063847659
> 2009-12-29 00:34:36,210 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:218)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> 2009-12-29 00:34:36,214 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020, call next(4170645244799815171, 1) from
> 192.168.1.108:53401: error: java.io.IOException:
> java.lang.IllegalArgumentException
> java.io.IOException: java.lang.IllegalArgumentException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:859)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1965)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> Caused by: java.lang.IllegalArgumentException
> at java.nio.Buffer.position(Buffer.java:218)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1114)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:58)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:189)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:106)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.nextInternal(HRegion.java:1776)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:1719)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1944)
> ... 5 more
> Looks like this bug was encountered before at
> https://issues.apache.org/jira/browse/HBASE-1495 and spanned a few JIRAs.
> It's supposed to be resolved as of 0.20.0, but we're running 0.20.2 and it
> took down one of our regionservers.
> I'm also attaching more of the log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
