Hi Shuja, This question is mentioned in the HBase FAQ, here:
http://wiki.apache.org/hadoop/Hbase/FAQ_Operations#A3 which points to the HBase book: http://hbase.apache.org/book.html#ulimit "HBase is a database. It uses a lot of files all at the same time. The default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is insufficient (On mac os x its 256). Any significant amount of loading will lead you to "FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?" ... Do yourself a favor and change the upper bound on the number of file descriptors. Set it to north of 10k. See the above referenced FAQ for how. You should also up the hbase users' nproc setting; under load, a low-nproc setting could manifest as OutOfMemoryError. To be clear, upping the file descriptors and nproc for the user who is running the HBase process is an operating system configuration, not an HBase configuration. Also, a common mistake is that administrators will up the file descriptors for a particular user but for whatever reason, HBase will be running as some one else. HBase prints in its logs as the first line the ulimit its seeing. Ensure its correct." - Ian On Aug 9, 2011, at 10:32 AM, Shuja Rehman wrote: Hi All, I am running map reduce jobs in queue and after some jobs, the following exception starts to come. Let me know if I need to change any settings or anything else Thanks in advance. 11/08/09 06:19:56 INFO mapreduce.TableOutputFormat: Created table instance for mytable 11/08/09 06:19:56 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.0.3.85:2181 sessionTimeout=180000 watcher=hconnection 11/08/09 06:19:56 ERROR mapreduce.TableInputFormat: org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException: Too many open files at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:941) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) at org.apache.hadoop.mapreduce.Job.submit(Job.java:476) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506) at hbaseaggregator.AggregatorRunner.runJob(AggregatorRunner.java:242) at hbaseaggregator.AggregatorRunner.Aggregator(AggregatorRunner.java:180) at hbaseaggregator.AggregatorDriver.main(AggregatorDriver.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Caused by: java.io.IOException: Too many open files at sun.nio.ch.IOUtil.initPipe(Native Method) at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at java.nio.channels.Selector.open(Selector.java:209) at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:160) at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:331) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:377) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:97) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:119) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989) ... 27 more 11/08/09 06:19:56 INFO mapred.JobClient: Cleaning up the staging area hdfs:// hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383<http://hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383> Exception in thread "main" java.lang.InternalError at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:755) at sun.misc.URLClassPath.getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run(URLClassLoader.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:296) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:435) at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2289) at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328) at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282) at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224) at java.util.ResourceBundle.getBundle(ResourceBundle.java:705) at java.util.logging.Level.getLocalizedName(Level.java:223) at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:64) at java.util.logging.StreamHandler.publish(StreamHandler.java:179) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88) at java.util.logging.Logger.log(Logger.java:458) at java.util.logging.Logger.doLog(Logger.java:480) at java.util.logging.Logger.log(Logger.java:569) at hbaseaggregator.AggregatorDriver.main(AggregatorDriver.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:114) at java.util.jar.JarFile.<init>(JarFile.java:135) at java.util.jar.JarFile.<init>(JarFile.java:72) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:646) at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:540) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:607) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:599) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:753) ... 29 more -- Regards Shuja-ur-Rehman Baig <http://pk.linkedin.com/in/shujamughal>
