Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by AndrewPurtell: http://wiki.apache.org/hadoop/Hbase/Troubleshooting The comment on the change is: Added item 9: ZooKeeper SessionExpired events ------------------------------------------------------------------------------ 1. [#6 Problem: "No live nodes contain current block"] 1. [#7 Problem: DFS instability and/or regionserver lease timeouts] 1. [#8 Problem: Instability on Amazon EC2] + 1. [#9 Problem: Zookeeper SessionExpired events] [[Anchor(1)]] == 1. Problem: Master initializes, but Region Servers do not == @@ -126, +127 @@ * In accordance with your hardware, tune your heap space / garbage collector settings in the HBASE_OPTS variable of {{{$HBASE_CONF/hbase-env.sh}}}. Try the ''concurrent garbage collector'' {{{(-XX:+UseConcMarkSweepGC)}}} to avoid to stop the threads during GC. Read these articles for more info about Hotspot GC settings * [http://java.sun.com/docs/hotspot/gc1.4.2/faq.html Garbage collector FAQ] Quick overview * [http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html Tuning garbage collector in Java SE 6] - * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode }}} + * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelGCThreads=8 }}} [[Anchor(8)]] == 8. Problem: Instability on Amazon EC2 == @@ -141, +142 @@ * Use X-Large instances if you are also running mappers and reducers co-located with system daemons. * Consider splitting storage and computational function over disjoint instance sets. + [[Anchor(9)]] + == 9. Problem: ZooKeeper SessionExpired events == + * Master or RegionServers reinitialize their ZooKeeper wrappers after receiving SessionExpired events. + * Master or RegionServer ephemeral nodes disappear while the node is still otherwise functional. + === Causes === + * Java GC is starving the ZooKeeper heartbeat thread. + === Resolution === + * Increase the session timeout. For example, add the following to your hbase-site.xml to increase the timeout from the default of 10 seconds to 60 seconds. + {{{ + <property> + <name>zookeeper.session.timeout</name> + <value>60000</value> + </property> + }}} + * For Java SE 6, some users have had success with {{{ -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelGCThreads=8 }}} +
