Hi folks,
(Resending to this group, sent to common-dev before, pretty sure that's for
Hadoop internal development - sorry for that..)
I'm pretty stuck here. I've been researching for hours and I haven't made
any forward progress on this one.
I have a vmWare installation of Cloudera Hadoop 0.20. The following
commands to create a directory and copy a file from the shared folder *work
fine*, so I'm confident everything is setup correctly:
[cloudera@localhost bin]$ hadoop fs -mkdir /user/cloudera/testdir
[cloudera@localhost bin]$ hadoop fs -put /mnt/hgfs/shared_folder/file1.txt
/user/cloudera/testdir/file1.txt
The file shows up fine in the HDFS doing it this way on the Linux VM.
*However*, when I try doing the equivalent operation in Java everything
works great until I try to close() FSDataOutputStream.
I'm left with the new directory and a zero byte size file. One suspicious
thing is that the user is "admin" instead of "cloudera" which I haven't
figured out why. Here is the error:
12/05/19 09:45:46 INFO hdfs.DFSClient: Exception in createBlockOutputStream
127.0.0.1:50010 java.net.ConnectException: Connection refused: no further
information
12/05/19 09:45:46 INFO hdfs.DFSClient: Abandoning block
blk_1931357292676354131_1068
12/05/19 09:45:46 INFO hdfs.DFSClient: Excluding datanode 127.0.0.1:50010
12/05/19 09:45:46 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/admin/testdir/file1.txt could only be replicated to 0 nodes, instead
of 1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1533)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:667)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
There are certainly lots of search references to "*could only be replicated
to 0 nodes, instead of 1*" but chasing down those suggestions hasn't helped.
I have run *jps* and* netstat* and that looks good. All services are
running, all port seem to be good. The *health check* looks good, plenty
of disk space, no failed nodes...
Here is the java (it fails when it hits "fs.close()":
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class TestFileTrans {
public static void main(String[] args) {
Configuration config = new Configuration();
config.addResource(new
Path("c:/_bigdata/client_libs/core-site.xml"));
config.addResource(new
Path("c:/_bigdata/client_libs/hdfs-site.xml"));
System.out.println("hadoop.tmp.dir: " +
config.get("hadoop.tmp.dir"));
try{
FileSystem dfs = FileSystem.get(config);
// this will default to admin unless the workingDirectory is
explicitly set..
System.out.println("HDFS Working Directory: " +
dfs.getWorkingDirectory().toString());
String dirName = "testdir";
Path src = new Path(dfs.getWorkingDirectory()+"/"+dirName);
dfs.mkdirs(src);
System.out.println("HDFS Directory created: " +
dfs.getWorkingDirectory().toString());
loadFile(dfs, src);
}catch(IOException e){
System.out.println("Error" + e.getMessage());
}
}
private static void loadFile(FileSystem dfs, Path src) throws
IOException{
FileInputStream fis = new
FileInputStream("c:/_bigdata/shared_folder/file1.txt");
int len = fis.available();
byte[] btr = new byte[len];
fis.read(btr);
FSDataOutputStream fs = dfs.create(new Path(src.toString()
+"/file1.txt"));
fs.write(btr);
fs.flush();
fs.close();
}
}
Any help would be greatly appreciated!
