Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.
The following page has been changed by LohitVijayarenu: http://wiki.apache.org/lucene-hadoop/HadoopDfsReadWriteExample New page: ~+Simple Example to Read and Write files from Hadoop DFS+~ Reading from and writing to Hadoop DFS is no different from how it is done with other file systems. The below example reads a file from DFS and writes it to another file on DFS (copy command). Hadoop [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem] API describes the methods available to user. Let us walk through the code to understand how it is done. Create a [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem] instance by passing a new Configuration object. {{{ Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); }}} Given an input/output file name as string, we construct inFile/outFile Path objects. Most of the [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem] APIs accepts [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/Path.html Path] objects. {{{ Path inFile = new Path(argv[0]); Path outFile = new Path(argv[1]); }}} Validate the input/output paths before reading/writing. {{{ if (!fs.exists(inFile)) printAndExit("Input file not found"); if (!fs.isFile(inFile)) printAndExit("Input should be a file"); if (fs.exists(outFile)) printAndExit("Output already exists"); }}} Open inFile for reading. {{{ FSDataInputStream in = fs.open(inFile); }}} Open outFile for writing. {{{ FSDataOutputStream out = fs.create(outFile); }}} Read from input stream and write to output stream until EOF. {{{ while ((bytesRead = in.read(buffer)) > 0) { out.write(buffer, 0, bytesRead); } }}} Close the streams when done. {{{ in.close(); out.close(); }}}