You must not have set it everywhere. Its same issue as yesterday. St.Ack
On Wed, Aug 10, 2011 at 6:43 AM, Shuja Rehman <[email protected]> wrote: > Hi, > I have set the limits as explained in the following link. > > https://ccp.cloudera.com/display/CDHDOC/HBase+Installation#HBaseInstallation-SettingUserLimitsforHBase > > Now it works fine for some more jobs But after that, I am getting following > exception now. > > 11/08/10 06:33:07 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=10.0.3.85:2181 sessionTimeout=180000 watcher=hconnection > 11/08/10 06:33:07 INFO zookeeper.ClientCnxn: Opening socket connection to > server /10.0.3.85:2181 > 11/08/10 06:33:07 INFO zookeeper.ClientCnxn: Socket connection established > to hadoop.zoniversal.com/10.0.3.85:2181, initiating session > 11/08/10 06:33:07 INFO zookeeper.ClientCnxn: Session establishment complete > on server hadoop.zoniversal.com/10.0.3.85:2181, sessionid = > 0x131628a4a072891, negotiated timeout = 40000 > 11/08/10 06:33:07 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 11/08/10 06:33:07 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor > exception for block blk_156159297298887768_25457java.io.IOException: Too > many open files > at sun.nio.ch.IOUtil.initPipe(Native Method) > at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49) > at > sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:178) > at java.io.DataInputStream.readLong(DataInputStream.java:399) > at > org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2638) > > 11/08/10 06:33:07 WARN hdfs.DFSClient: DataStreamer Exception: > java.io.InterruptedIOException: Interruped while waiting for IO on channel > java.nio.channels.SocketChannel[connected local=/10.0.3.85:57316 remote=/ > 10.0.3.85:50010]. 484999 millis timeout left. > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:349) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) > at > org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2526) > > 11/08/10 06:33:07 INFO hdfs.DFSClient: Error Recovery for block > blk_156159297298887768_25457 waiting for responder to exit. > 11/08/10 06:33:07 WARN hdfs.DFSClient: Error Recovery for block > blk_156159297298887768_25457 bad datanode[0] 10.0.3.85:50010 > 11/08/10 06:33:07 INFO mapred.JobClient: Cleaning up the staging area > hdfs:// > hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0619 > Aug 10, 2011 6:33:08 AM hbaseaggregator.AggregatorDriver main > SEVERE: null > java.io.IOException: All datanodes 10.0.3.85:50010 are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2774) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477) > 11/08/10 06:33:08 ERROR hdfs.DFSClient: Exception closing file > /var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0619/libjars/zookeeper-3.3.3-cdh3u0.jar > : java.io.IOException: All datanodes 10.0.3.85:50010 are bad. Aborting... > java.io.IOException: All datanodes 10.0.3.85:50010 are bad. Aborting... > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2774) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477) > > > kindly let me know the solution. > thnx > > On Tue, Aug 9, 2011 at 8:38 PM, Ian Varley <[email protected]> wrote: > >> Hi Shuja, >> >> This question is mentioned in the HBase FAQ, here: >> >> http://wiki.apache.org/hadoop/Hbase/FAQ_Operations#A3 >> >> which points to the HBase book: >> >> http://hbase.apache.org/book.html#ulimit >> >> "HBase is a database. It uses a lot of files all at the same time. The >> default ulimit -n -- i.e. user file limit -- of 1024 on most *nix systems is >> insufficient (On mac os x its 256). Any significant amount of loading will >> lead you to "FAQ: Why do I see "java.io.IOException...(Too many open files)" >> in my logs?" ... Do yourself a favor and change the upper bound on the >> number of file descriptors. Set it to north of 10k. See the above referenced >> FAQ for how. You should also up the hbase users' nproc setting; under load, >> a low-nproc setting could manifest as OutOfMemoryError. To be clear, upping >> the file descriptors and nproc for the user who is running the HBase process >> is an operating system configuration, not an HBase configuration. Also, a >> common mistake is that administrators will up the file descriptors for a >> particular user but for whatever reason, HBase will be running as some one >> else. HBase prints in its logs as the first line the ulimit its seeing. >> Ensure its correct." >> >> - Ian >> >> On Aug 9, 2011, at 10:32 AM, Shuja Rehman wrote: >> >> Hi All, >> >> I am running map reduce jobs in queue and after some jobs, the following >> exception starts to come. Let me know if I need to change any settings or >> anything else >> >> Thanks in advance. >> >> >> 11/08/09 06:19:56 INFO mapreduce.TableOutputFormat: Created table instance >> for mytable >> 11/08/09 06:19:56 INFO zookeeper.ZooKeeper: Initiating client connection, >> connectString=10.0.3.85:2181 sessionTimeout=180000 watcher=hconnection >> 11/08/09 06:19:56 ERROR mapreduce.TableInputFormat: >> org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException: >> Too many open files >> at >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:991) >> at >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:302) >> at >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:293) >> at >> >> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:156) >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167) >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145) >> at >> >> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:91) >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) >> at >> >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) >> at >> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:941) >> at >> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) >> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) >> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) >> at >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833) >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:476) >> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506) >> at >> hbaseaggregator.AggregatorRunner.runJob(AggregatorRunner.java:242) >> at >> hbaseaggregator.AggregatorRunner.Aggregator(AggregatorRunner.java:180) >> at hbaseaggregator.AggregatorDriver.main(AggregatorDriver.java:60) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:186) >> Caused by: java.io.IOException: Too many open files >> at sun.nio.ch.IOUtil.initPipe(Native Method) >> at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49) >> at >> >> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) >> at java.nio.channels.Selector.open(Selector.java:209) >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:160) >> at org.apache.zookeeper.ClientCnxn.<init>(ClientCnxn.java:331) >> at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:377) >> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:97) >> at >> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:119) >> at >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:989) >> ... 27 more >> >> 11/08/09 06:19:56 INFO mapred.JobClient: Cleaning up the staging area >> hdfs:// >> >> hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383 >> < >> http://hadoop.zoniversal.com/var/lib/hadoop-0.20/cache/mapred/mapred/staging/root/.staging/job_201107251124_0383 >> > >> Exception in thread "main" java.lang.InternalError >> at >> sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:755) >> at sun.misc.URLClassPath.getResource(URLClassPath.java:169) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:194) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at sun.misc.Launcher$ExtClassLoader.findClass(Launcher.java:229) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:296) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> at >> java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:435) >> at >> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2289) >> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364) >> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328) >> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282) >> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224) >> at java.util.ResourceBundle.getBundle(ResourceBundle.java:705) >> at java.util.logging.Level.getLocalizedName(Level.java:223) >> at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:64) >> at java.util.logging.StreamHandler.publish(StreamHandler.java:179) >> at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88) >> at java.util.logging.Logger.log(Logger.java:458) >> at java.util.logging.Logger.doLog(Logger.java:480) >> at java.util.logging.Logger.log(Logger.java:569) >> at hbaseaggregator.AggregatorDriver.main(AggregatorDriver.java:72) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:186) >> Caused by: java.util.zip.ZipException: error in opening zip file >> at java.util.zip.ZipFile.open(Native Method) >> at java.util.zip.ZipFile.<init>(ZipFile.java:114) >> at java.util.jar.JarFile.<init>(JarFile.java:135) >> at java.util.jar.JarFile.<init>(JarFile.java:72) >> at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:646) >> at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:540) >> at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:607) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:599) >> at >> sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:753) >> ... 29 more >> >> -- >> Regards >> Shuja-ur-Rehman Baig >> <http://pk.linkedin.com/in/shujamughal> >> >> > > > -- > Regards > Shuja-ur-Rehman Baig > <http://pk.linkedin.com/in/shujamughal> >
