What actually happened in brick4 (the node that refused r910(master)'s
connection) is that it
" failed to remove existing native library file: /tmp/
snappy-1.0.4.1-libsnappyjava.so
at
org.xerial.snappy.SnappyLoader.extractLibraryFile(SnappyLoader.java:376)"
then
java.lang.NullPointerException
at
org.apache.hama.bsp.message.compress.SnappyCompressor.compressBundle(SnappyCompressor.java:56)
But I actually run in the Hama directory and snappy-java-1.0.4.1.jar is
included in hama/lib dir.
I even include the hama/lib/snappy-java-1.0.4.1.jar in CLASSPATH.
What happened?
2012/9/11 Sandy Ding <[email protected]>
> Hi, all,
>
> I newly set up a 3-node hama cluster following the HamaInstallationGuide.
> But I got some confusing errors when running the pi examples, the tasklog
> is as follows:
>
> 12/09/11 19:23:39 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=brick4:
> 21810,r910:21810 sessionTimeout=1200000
> watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImp
> l@e61a35
> 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Opening socket connection to
> server brick4/10.131.
> 201.14:21810
> 12/09/11 19:23:39 INFO sync.ZooKeeperSyncClientImpl: Start connecting to
> Zookeeper! At r910.ppi
> /10.131.201.90:61001
> 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Socket connection established
> to brick4/10.131.201
> .14:21810, initiating session
> 12/09/11 19:23:39 INFO zookeeper.ClientCnxn: Session establishment
> complete on server brick4/10
> .131.201.14:21810, sessionid = 0x139b47286bf000c, negotiated timeout =
> 1200000
> 12/09/11 19:23:40 INFO ipc.NettyTransceiver: Connecting to brick4.r715/
> 10.131.201.14:61003
> 12/09/11 19:23:40 INFO ipc.NettyTransceiver: [id: 0x0118278a] OPEN
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClien
> tSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedK
> eys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSoc
> ketPipelineSink.java:276)
> at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 12/09/11 19:23:40 INFO ipc.NettyTransceiver: [id: 0x0118278a] CLOSED
> 12/09/11 19:23:40 INFO ipc.NettyTransceiver: Remote peer brick4.r715/
> 10.131.201.14:61003 closed
> connection.
> 12/09/11 19:23:40 ERROR bsp.BSPTask: Error running bsp setup and bsp
> function.
> java.io.IOException: Error connecting to brick4.r715/10.131.201.14:61003
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91)
> at
> org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.j
> ava:83)
> at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328)
> at
> org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 12/09/11 19:23:40 WARN ipc.NettyTransceiver: Unexpected exception from
> downstream.
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 12/09/11 19:23:40 INFO zookeeper.ZooKeeper: Session: 0x139b47286bf000c
> closed
> 12/09/11 19:23:40 INFO zookeeper.ClientCnxn: EventThread shut down
> 12/09/11 19:23:40 ERROR bsp.BSPTask: Shutting down ping service.
> 12/09/11 19:23:40 FATAL bsp.GroomServer: Error running child
> java.io.IOException: Error connecting to brick4.r715/10.131.201.14:61003
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91)
> at
> org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.java:83)
> at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328)
> at
> org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> java.io.IOException: Error connecting to brick4.r715/10.131.201.14:61003
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:163)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:128)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:91)
> at
> org.apache.hama.bsp.message.AvroMessageManagerImpl.transfer(AvroMessageManagerImpl.java:83)
> at org.apache.hama.bsp.BSPPeerImpl.sync(BSPPeerImpl.java:328)
> at
> org.apache.hama.examples.PiEstimator$MyEstimator.bsp(PiEstimator.java:69)
> at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:166)
> at org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
> at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1158)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:384)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:354)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:276)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> .....
>
> Can anybody help? I am really desparate...
>
> Thanks in advance,
>
> Sandy
>
>
