I have a single node hadoop cluster setup behind a firewall and am trying to
create files using a java program outside the firewall and get the exception
below. The java program works fine inside the firewall. The ip address for the
single cluster is xxx.xxx.xxx.xxx however it appears that in the
createBlockOutputStream the client things the data node is at ip
yyy.yyy.yyy.yyy (the internal address of the cluster) which is not accessible.
The java code looks like this (using hadoop 1.1.2):
private static void createHdfsFile() throws IOException {
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://"+hdfsHost+":9000");
FileSystem hdfs = FileSystem.get(conf);
System.out.println("HDFS Working Directory: " +
hdfs.getWorkingDirectory().toString());
FSDataOutputStream os = hdfs.create(new Path("/user/hadoop/test2.txt"));
os.writeChars("Example text\n for a hadoop write call\n\ntesting\n");
os.close();
}
Any idea how I can get this to work?
HDFS Working Directory: hdfs://xxx.xxx.xxx.xxx:9000/user/zzzzz
Jun 26, 2013 8:08:52 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
createBlockOutputStream
INFO: Exception in createBlockOutputStream yyy.yyy.yyy.yyy:50010
java.net.ConnectException: Connection timed out: no furth
er information
Jun 26, 2013 8:08:52 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
nextBlockOutputStream
INFO: Abandoning block blk_4933973859208379842_1028
Jun 26, 2013 8:08:53 PM org.apache.hadoop.hdfs.DFSClient$DFSOutputStream
nextBlockOutputStream
INFO: Excluding datanode yyy.yyy.yyy.yyy:50010
Jun 26, 2013 8:08:53 PM
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer run
WARNING: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File /user/hadoop/test2.txt
could only be replicated to 0 nodes, instead of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1558)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at $Proxy1.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at $Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989)
