[
https://issues.apache.org/jira/browse/HBASE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611548#action_12611548
]
Jean-Daniel Cryans commented on HBASE-706:
------------------------------------------
I added a reservation linked list that takes a number of blocks of 5M defined
by configuration and by default of 4. Using a HRegionServer that works like
OOMERegionServer (was easier I thought), the loader.jsp in attachment and the
reservation list, here is a trace of a OOME that is correctly handled :
{noformat}
2008-07-07 17:51:42,853 DEBUG org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 1705462391/contents store size is 23.7m
2008-07-07 17:51:42,868 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region loading,53841,1215463619617 in 6sec
2008-07-07 17:52:23,425 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Ran out of memory
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2760)
at java.util.Arrays.copyOf(Arrays.java:2734)
at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
at java.util.ArrayList.add(ArrayList.java:351)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1152)
...
2008-07-07 17:52:23,587 DEBUG org.apache.hadoop.hbase.RegionHistorian: Offlined
2008-07-07 17:52:23,588 INFO org.apache.hadoop.ipc.Server: Stopping server on
60020
2008-07-07 17:52:23,588 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0
on 60020: exiting
2008-07-07 17:52:23,588 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1
on 60020: exiting
2008-07-07 17:52:23,588 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2
on 60020: exiting
2008-07-07 17:52:23,588 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 60020: exiting
...
2008-07-07 17:52:33,431 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2008-07-07 17:52:33,432 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer:
regionserver/0:0:0:0:0:0:0:0:60020 exiting
2008-07-07 17:52:33,432 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown thread.
2008-07-07 17:52:33,432 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread complete
{noformat}
> On OOME, regionserver sticks around and doesn't go down with cluster
> --------------------------------------------------------------------
>
> Key: HBASE-706
> URL: https://issues.apache.org/jira/browse/HBASE-706
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jean-Daniel Cryans
> Fix For: 0.2.0
>
>
> On John Gray cluster, an errant, massive, store file caused us OOME.
> Shutdown of cluster left this regionserver in place. A thread dump failed
> with OOME. Here is last thing in log:
> {code}
> 2008-06-25 03:21:55,111 INFO org.apache.hadoop.hbase.HRegionServer: worker
> thread exiting
> 2008-06-25 03:24:26,923 FATAL org.apache.hadoop.hbase.HRegionServer: Set stop
> flag in regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.<init>(HashMap.java:226)
> at java.util.HashSet.<init>(HashSet.java:103)
> at
> org.apache.hadoop.hbase.HRegionServer.getRegionsToCheck(HRegionServer.java:1789)
> at
> org.apache.hadoop.hbase.HRegionServer$Flusher.enqueueOptionalFlushRegions(HRegionServer.java:479)
> at
> org.apache.hadoop.hbase.HRegionServer$Flusher.run(HRegionServer.java:385)
> 2008-06-25 03:24:26,923 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 2 on 60020, call batchUpdate(items,,1214272763124, 9223372036854775807,
> [EMAIL PROTECTED]) from 192.168.249.230:38278: error: java.io.IOException:
> Server not running
> java.io.IOException: Server not running
> at
> org.apache.hadoop.hbase.HRegionServer.checkOpen(HRegionServer.java:1758)
> at
> org.apache.hadoop.hbase.HRegionServer.batchUpdate(HRegionServer.java:1547)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:901)
> {code}
> If I get an OOME just trying to threaddump, would seem to indicate we need to
> start keeping a little memory resevoir around for emergencies such as this
> just so we can shutdown clean.
> Moving this into 0.2. Seems important to fix if robustness is name of the
> game.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
