If you don't have additional stack information it is difficult to guess. According to http://docs.oracle.com/javase/7/docs/api/java/io/File.html#delete() it should throw an IOException that contains the underlying error. If the shared object can't be deleted, I would try to start by checking rights or what on the other machines is not like on the machine that failed.
2012/9/11 Sandy Ding <[email protected]> > Any possible reasons for that? > > 2012/9/11 Thomas Jungblut <[email protected]> > > > Hi, > > > > that is actually a good question. > > According to the exception and the code [1] it was unable to delete the > > file. > > Actually this could have various reasons, what operating system are you > > running? > > > > [1] > > > > > https://github.com/xerial/snappy-java/blob/develop/src/main/java/org/xerial/snappy/SnappyLoader.java#L374 > > > > > > 2012/9/11 Sandy Ding <[email protected]> > > > > > What actually happened in brick4 (the node that refused r910(master)'s > > > connection) is that it > > > " failed to remove existing native library file: /tmp/ > > > snappy-1.0.4.1-libsnappyjava.so > > > at > > > > org.xerial.snappy.SnappyLoader.extractLibraryFile(SnappyLoader.java:376)" > > > then > > > java.lang.NullPointerException > > > at > > > > > > > > > org.apache.hama.bsp.message.compress.SnappyCompressor.compressBundle(SnappyCompressor.java:56) > > > > > > But I actually run in the Hama directory and snappy-java-1.0.4.1.jar is > > > included in hama/lib dir. > > > I even include the hama/lib/snappy-java-1.0.4.1.jar in CLASSPATH. > > > What happened? > > > > > > > > > 2012/9/11 Sandy Ding <[email protected]> > > > > > > > Hi, all, > > > > > > > > I newly set up a 3-node hama cluster following the > > HamaInstallationGuide. > > > > But I got some confusing errors when running the pi examples, the > > tasklog > > > > is as follows: > > > > > > > > 12/09/11 19:23:39 INFO zookeeper.ZooKeeper: Initiating client > > connection, > > > > connectString=brick4: > > > > 21810,r910:21810 sessionTimeout=1200000 > > > > watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImp > > > > l@e61a35 > > > > 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Opening socket > connection > > to > > > > server brick4/10.131. > > > > 201.14:21810 > > > > 12/09/11 19:23:39 INFO sync.ZooKeeperSyncClientImpl: Start connecting > > to > > > > Zookeeper! At r910.ppi > > > > /10.131.201.90:61001 > > > > 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Socket connection > > > established > > > > to brick4/10.131.201 > > > > .14:21810, initiating session > > > > 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Session establishment > > > > complete on server brick4/10 > > > > .131.201.14:21810, sessionid = 0x139b47286bf000c, negotiated timeout > = > > > > 1200000 > > > > 12/09/11 19:23:40 INFO ipc.NettyTransceiver: Connecting to > brick4.r715/ > > > > 10.131.201.14:61003 > > > > 12/09/11 19:23:40 INFO ipc.NettyTransceiver: [id: 0x0118278a] OPEN > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClien > > > > tSocketPipelineSink.java:384) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedK > > > > eys(NioClientSocketPipelineSink.java:354) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSoc > > > > ketPipelineSink.java:276) > > > > at > > > > > > > > > > org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > > > > at > > > > > > > > > > org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 12/09/11 19:23:40 INFO ipc.NettyTransceiver: [id: 0x0118278a] CLOSED > > > > 12/09/11 19:23:40 INFO ipc.NettyTransceiver: Remote peer brick4.r715/ > > > > 10.131.201.14:61003 closed > > > > connection. > > > > 12/09/11 19:23:40 ERROR bsp.BSPTask: Error running bsp setup and bsp > > > > function. > > > > java.io.IOException: Error connecting to brick4.r715/ > > 10.131.201.14:61003 > > > > at > > > > > > > > > > org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163) > > > > at > > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128) > > > > at > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91) > > > > at > > > > > > > > > > org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.j > > > > ava:83) > > > > at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328) > > > > at > > > > > > org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69) > > > > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166) > > > > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) > > > > at > > > > > > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) > > > > Caused by: java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 12/09/11 19:23:40 WARN ipc.NettyTransceiver: Unexpected exception > from > > > > downstream. > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > 12/09/11 19:23:40 INFO zookeeper.ZooKeeper: Session: > 0x139b47286bf000c > > > > closed > > > > 12/09/11 19:23:40 INFO zookeeper.ClientCnxn: EventThread shut down > > > > 12/09/11 19:23:40 ERROR bsp.BSPTask: Shutting down ping service. > > > > 12/09/11 19:23:40 FATAL bsp.GroomServer: Error running child > > > > java.io.IOException: Error connecting to brick4.r715/ > > 10.131.201.14:61003 > > > > at > > > > > > > > > > org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163) > > > > at > > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128) > > > > at > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91) > > > > at > > > > > > > > > > org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.java:83) > > > > at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328) > > > > at > > > > > > org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69) > > > > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166) > > > > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) > > > > at > > > > > > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) > > > > Caused by: java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > > java.io.IOException: Error connecting to brick4.r715/ > > 10.131.201.14:61003 > > > > at > > > > > > > > > > org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163) > > > > at > > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128) > > > > at > > > > org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91) > > > > at > > > > > > > > > > org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.java:83) > > > > at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328) > > > > at > > > > > > org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69) > > > > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166) > > > > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143) > > > > at > > > > > > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158) > > > > Caused by: java.net.ConnectException: Connection refused > > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > > > at > > > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354) > > > > at > > > > > > > > > > org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276) > > > > at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > ..... > > > > > > > > Can anybody help? I am really desparate... > > > > > > > > Thanks in advance, > > > > > > > > Sandy > > > > > > > > > > > > > >
