Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for 
change notification.

The following page has been changed by LohitVijayarenu:
http://wiki.apache.org/lucene-hadoop/HadoopDfsReadWriteExample

New page:
~+Simple Example to Read and Write files from Hadoop DFS+~

Reading from and writing to Hadoop DFS is no different from how it is done with 
other file systems. The below example reads a file from DFS and writes it to 
another file on DFS (copy command). 

Hadoop 
[http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html 
FileSystem] API describes the methods available to user. Let us walk through 
the code to understand how it is done.

Create a 
[http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html 
FileSystem] instance by passing a new Configuration object. 

{{{
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
}}}

Given an input/output file name as string, we construct inFile/outFile Path 
objects. Most of the 
[http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html 
FileSystem] APIs accepts 
[http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/Path.html Path] 
objects.

{{{
Path inFile = new Path(argv[0]);
Path outFile = new Path(argv[1]);
}}}

Validate the input/output paths before reading/writing.

{{{
if (!fs.exists(inFile))
  printAndExit("Input file not found");
if (!fs.isFile(inFile))
  printAndExit("Input should be a file");
if (fs.exists(outFile))
  printAndExit("Output already exists");
}}}


Open inFile for reading.

{{{
FSDataInputStream in = fs.open(inFile);
}}}

Open outFile for writing.

{{{
FSDataOutputStream out = fs.create(outFile);
}}}

Read from input stream and write to output stream until EOF.

{{{
while ((bytesRead = in.read(buffer)) > 0) {
  out.write(buffer, 0, bytesRead);
}
}}}

Close the streams when done.

{{{
in.close();
out.close();
}}}

Reply via email to