Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by JeanAdrien: http://wiki.apache.org/hadoop/Hbase/Troubleshooting The comment on the change is: Added JVM GC settings advise ------------------------------------------------------------------------------ == 7. Problem: DFS instability and/or regionserver lease timeouts == * HBase regionserver leases expire during start up * HBase daemons cannot find block locations in HDFS during start up or other periods of load + * HBase regionserver restarts after beeing unable to report to master: + {{{ + 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten times longer than scheduled: 10000 + 2009-02-24 10:01:33,516 WARN org.apache.hadoop.hbase.util.Sleeper: We slept xxx ms, ten times longer than scheduled: 15000 + 2009-02-24 10:01:36,472 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: unable to report to master for xxx milliseconds - retrying + }}} === Causes === * Excessive connection establishment latency (HRPC sets up connections on demand) * Slow host name resolution * Network bandwidth overcommitment + * Very long garbage collector task for the RegionServer JVM: The ''default garbage collector'' of the HotspotTM JavaTM Virtual Machine runs a ''full gc'' on the ''old space'' when the memory space is full, which can represent about 90% of the allocated heap space. During this task, the running program is stopped, the timers as well. If the heap space is mostly in the swap partition, and moreover if it is larger than the physical memory, the subsequent swap can yield to I/O overload and takes several minutes. === Resolution === * Insure that host name resolution latency is low, or use static entries in /etc/hosts * Monitor the network and insure that adequate bandwidth is available for HRPC transactions + * In accordance with your hardware, tune your heap space / garbage collector settings in the HBASE_OPTS variable of {{{$HBASE_CONF/hbase-env.sh}}}. Try the ''concurrent garbage collector'' {{{(-XX:+UseConcMarkSweepGC)}}} to avoid to stop the threads during GC. Read these articles for more info about Hotspot GC settings + * [http://java.sun.com/docs/hotspot/gc1.4.2/faq.html Garbage collector FAQ] Quick overview + * [http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html Tuning garbage collector in Java SE 6]
