Dear List,
I'm using Hadoop in a Multi-homed environment. Basically the Namenode, and
Datanodes are connected via a special network for datatransfer 10.xxx.xxx.xxx.
I installed the Hadoop tool on a computer which can access the nodes in the
hadoop cluster via a second network 192.168.xxx.xxx .
I want to use this computer to copy data into HDFS. However, all operations
which try to copy data directly onto the datanodes are failing.
Basically I can do ls, mkdir and even copy empty files, however, commands like:
hadoop fs -put d:/temp/* hdfs://192.168.<namenode>/user/<me>/to_load/
are failing.
As you can see in the hadoop tool output below the client is trying to access
the datanodes via the IP addresses from the datatransfer network and not via
the public second network.
The strange thing in the configuration files on the namenode the parameter
dfs.client.use.datanode.hostname is set to true. From my untestanding I,
therefore, shouldn't see the logline
15/01/19 13:51:11 DEBUG hdfs.DFSClient: pipeline = 10.x.x.13:50010
At all
thanks in advance,
Ingo Thon
Output from hadoop command
15/01/19 13:51:11 DEBUG ipc.Client: IPC Client (7749777) connection to
/192.168.xxx.xxx:8020 from me sending #12
15/01/19 13:51:11 DEBUG ipc.Client: IPC Client (7749777) connection to
/192.168.xxx.xxx:8020 from thon_i got value #12
15/01/19 13:51:11 DEBUG ipc.ProtobufRpcEngine: Call: addBlock took 0ms
15/01/19 13:51:11 DEBUG hdfs.DFSClient: pipeline = 10.x.x.13:50010
15/01/19 13:51:11 DEBUG hdfs.DFSClient: Connecting to datanode 10.x.x.13:50010
15/01/19 13:51:21 DEBUG ipc.Client: IPC Client (7749777) connection to
/192.168.xxx.xxx:8020 from thon_i: closed
15/01/19 13:51:21 DEBUG ipc.Client: IPC Client (7749777) connection to
/192.168.xxx.xxx:8020 from thon_i: stopped, remaining connections 0
15/01/19 13:51:32 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at
org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1526)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1328)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)
15/01/19 13:51:32 INFO hdfs.DFSClient: Abandoning
BP-20yyyyyyy26-10.x.x.x-1415yyyyy790:blk_1074387723_646941
15/01/19 13:51:32 DEBUG ipc.Client: The ping interval is 60000 ms.
