After doing some google researches, it seems that your machine is running out of memory when the RawLocalFilesystem forks in order to create and chmod a new file (since that creates a new process with as much mem as the original process).
It was discussed a lot in https://issues.apache.org/jira/browse/HADOOP-5059 since this is in Hadoop's land. Get more RAM, or more swap. J-D On Wed, Jul 7, 2010 at 10:53 AM, manua <[email protected]> wrote: > > Hi, > > I am using Hbase version 0.20.5. > > Also, I am running both the server on client on the same box. Please find > below the other log snippets, > > Log File : hbase-manu-regionserver-domU-12-31-39-06-62-43.log > > Wed Jul 7 13:42:17 EDT 2010 Starting regionserver on domU-12-31-39-06-62-43 > ulimit -n 1024 > 2010-07-07 13:42:18,222 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a distinct > region server because hbase.cluster.distributed is false > > Files hbase-manu-zookeeper-domU-12-31-39-06-62-43.out and > hbase-manu-regionserver-domU-12-31-39-06-62-43.out are blank. > > File:hbase-manu-zookeeper-domU-12-31-39-06-62-43.log > > 2010-07-07 13:42:15,799 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server > environment:java.library.path=/home/manu/hbase/hbase-0.20.5/bin/../lib/native/Linux-i386-32 > 2010-07-07 13:42:15,799 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:java.io.tmpdir=/tmp > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:java.compiler=<NA> > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:os.name=Linux > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:os.arch=i386 > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:os.version=2.6.21.7-2.ec2.v1.2.fc8xen > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:user.name=manu > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:user.home=/home/manu > 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer: > Server environment:user.dir=/home/manu/hbase/hbase-0.20.5 > 2010-07-07 13:42:15,843 INFO > org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot > /home/_hbase_/hbase-manu/zookeeper/version-2/snapshot.170 > 2010-07-07 13:42:15,879 INFO > org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 17f > 2010-07-07 13:42:17,090 INFO org.apache.zookeeper.server.NIOServerCnxn: > Connected to /127.0.0.1:36319 lastZxid 0 > 2010-07-07 13:42:17,091 INFO org.apache.zookeeper.server.NIOServerCnxn: > Creating new session 0x129ae00344b0000 > 2010-07-07 13:42:17,169 INFO org.apache.zookeeper.server.NIOServerCnxn: > Finished init of 0x129ae00344b0000 valid:true > 2010-07-07 13:42:17,258 INFO org.apache.zookeeper.server.NIOServerCnxn: > Connected to /127.0.0.1:36320 lastZxid 0 > 2010-07-07 13:42:17,258 INFO org.apache.zookeeper.server.NIOServerCnxn: > Creating new session 0x129ae00344b0001 > 2010-07-07 13:42:17,263 INFO org.apache.zookeeper.server.NIOServerCnxn: > Finished init of 0x129ae00344b0001 valid:true > 2010-07-07 13:42:18,437 WARN > org.apache.zookeeper.server.PrepRequestProcessor: Got exception when > processing sessionid:0x129ae00344b0000 type:create cxid:0xd > zxid:0xfffffffffffffffe txntype:unknown n/a > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > NodeExists > at > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) > at > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) > 2010-07-07 13:44:02,841 INFO org.apache.zookeeper.server.NIOServerCnxn: > Connected to /127.0.0.1:36325 lastZxid 0 > 2010-07-07 13:44:02,841 INFO org.apache.zookeeper.server.NIOServerCnxn: > Creating new session 0x129ae00344b0002 > 2010-07-07 13:44:02,850 INFO org.apache.zookeeper.server.NIOServerCnxn: > Finished init of 0x129ae00344b0002 valid:true > 2010-07-07 13:46:05,328 INFO > org.apache.zookeeper.server.PrepRequestProcessor: Processed session > termination request for id: 0x129ae00344b0001 > 2010-07-07 13:46:05,344 INFO org.apache.zookeeper.server.NIOServerCnxn: > closing session:0x129ae00344b0001 NIOServerCnxn: > java.nio.channels.SocketChannel[connected local=/127.0.0.1:2181 > remote=/127.0.0.1:36320] > > File : hbase-manu-master-domU-12-31-39-06-62-43.out > > java.io.IOException: Cannot run program "chmod": java.io.IOException: > error=12, Cannot allocate memory > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) > at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) > at org.apache.hadoop.util.Shell.run(Shell.java:134) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286) > at org.apache.hadoop.util.Shell.execCommand(Shell.java:354) > at org.apache.hadoop.util.Shell.execCommand(Shell.java:337) > at > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481) > at > org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473) > at > org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280) > at > org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:372) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484) > at > org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:843) > at > org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:393) > at > org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:284) > at org.apache.hadoop.hbase.regionserver.HLog$1.run(HLog.java:1055) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: java.io.IOException: error=12, Cannot > allocate memory > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) > at java.lang.ProcessImpl.start(ProcessImpl.java:65) > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) > ... 17 more > > Please let me know your inputs. > > Thanks, > Manu > > -- > View this message in context: > http://old.nabble.com/Java-Client-hangs-after-around-2-min-tp29097955p29099055.html > Sent from the HBase User mailing list archive at Nabble.com. > >
