Hi,
I wrote a short Program on Hadoop Reading and Writing. I want to read a
local file and write it to the hadoop.
The code is below:
public class HadoopDFSFileReadWrite {
public static void main(String[] argv) throws IOException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
FileSystem dfs = null;
try {
dfs = FileSystem.get(new URI("hdfs://root:[EMAIL PROTECTED]:50010"), conf);
//????? can not connect to datanode Exception Below:
/*
Exception in thread "main" java.net.SocketTimeoutException: timed out
waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:484)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:269)
at org.apache.hadoop.dfs.DFSClient.createNamenode(DFSClient.java:147)
at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:161)
at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(
DistributedFileSystem.java:65)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:159)
at readWrite.HadoopDFSFileReadWrite.main(HadoopDFSFileReadWrite.java:50)
*/
} catch (URISyntaxException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
if (argv.length != 2)
usage();
// Hadoop DFS deals with Path
Path inFile = new Path(argv[0]);
Path outFile = new Path(argv[1]);
// Check if input/output are valid
if (!fs.exists(inFile))
printAndExit("Input file not found");
if (!fs.isFile(inFile))
printAndExit("Input should be a file");
if (dfs.exists(outFile))
printAndExit("Output already exists");
// Read from and write to new file
FSDataInputStream in = fs.open(inFile);
FSDataOutputStream out = dfs.create(outFile);
byte buffer[] = new byte[256];
try {
System.out.println("Start...");
int bytesRead = 0;
while ((bytesRead = in.read(buffer)) > 0) {
out.write(buffer, 0, bytesRead);
}
System.out.println("End.....");
} catch (IOException e) {
System.out.println("Error while copying file");
} finally {
in.close();
out.close();
}
}
}
I wonder whether the rpc port is wrong? or there are other problems?
or Someone can give me a example to write to the hdfs?
Thanks,
Ryan