Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "Hbase/Troubleshooting" page has been changed by AndrewPurtell. http://wiki.apache.org/hadoop/Hbase/Troubleshooting?action=diff&rev1=35&rev2=36 -------------------------------------------------- <<Anchor(13)>> - == 13. Problem: Long client pauses under high load; or deadlock if using transactional HBase (THBase)== + == 13. Problem: Long client pauses under high load; or deadlock if using transactional HBase (THBase) == * Under high load, some client operations take a long time; waiting appears uneven * If using THBase, apparent deadlocks: for example, in thread dumps IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex() - == Causes == + === Causes === * The default number of regionserver RPC handlers is insufficient. - == Resolution == + === Resolution === * Increase the value of "hbase.regionserver.handler.count" in hbase-site.xml. The default is 10. Try 100. @@ -222, +222 @@ closing session 0x0 to sun.nio.ch.selectionkeyi...@656dc861 java.net.ConnectException: Connection refused }}} - == Causes == + === Causes === * Security group policy is blocking the Zookeeper port on a public address. - == Resolution == + === Resolution === * Use the internal EC2 host names when configuring the Zookeeper quorum peer list. <<Anchor(15)>> == 15. Problem: General operating environment issues -- zookeeper session timeouts, regionservers shutting down, etc == - == Causes == + === Causes === Various. - == Resolution == + === Resolution === See the [[http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting ZooKeeper Operating Environment Troubleshooting]] page. It has suggestions and tools for checking disk and networking performance; i.e. the operating environment your zookeeper and hbase are running in. ZooKeeper is the cluster's "canary". It'll be the first to notice issues if any so making sure its happy is the short-cut to a humming cluster. <<Anchor(16)>> == 16. Problem: Scanner performance is low == - == Causes == + === Causes === Default scanner caching (prefetching) is set to 1. The default is low because if a job takes too long processing, a scanner can time out, which causes unhappy jobs/people/emails. See item #10 above. - == Resolution == + === Resolution === * Increase the amount of prefetching on the scanner, to 10, or 100, or 1000, as appropriate for your workload: [[http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/client/HTable.html#scannerCaching|HTable.scannerCaching]] * This change can be accomplished globally by setting the hbase.client.scanner.caching property in hbase-site.xml to the desired value. <<Anchor(17)>> == 17. Problem: My shell or client application throws lots of scary exceptions during normal operation == - == Causes == + === Causes === Since 0.20.0 the default log level for org.apache.hadoop.hbase.* is DEBUG. - == Resolution == + === Resolution === On your clients, edit $HBASE_HOME/conf/log4j.properties and change this: {{{log4j.logger.org.apache.hadoop.hbase=DEBUG}}} to this: {{{log4j.logger.org.apache.hadoop.hbase=INFO}}}, or even {{{log4j.logger.org.apache.hadoop.hbase=WARN}}} .
