Thanks Ted. It solves my problem. :)
Ted Dunning-3 wrote:
>
>
> I think that file names of the form "file://directory-path" should work to
> give you local file access using this program.
>
>
> On 5/6/08 3:34 PM, "Ajey Shah" <[EMAIL PROTECTED]> wrote:
>
>>
>> Thanks Suresh. But even this program reads and writes from the HDFS. What
>> i
>> need to do is read from my normal local linux harddrive and write to the
>> HDFS.
>>
>> I'm sorry if I misunderstood your program.
>>
>> Thanks for replying. :)
>>
>>
>>
>> Babu, Suresh wrote:
>>>
>>>
>>> Try this program. Modify the HDFS configuration, if it is different from
>>> the default.
>>>
>>> import java.io.File;
>>> import java.io.IOException;
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.FileStatus;
>>> import org.apache.hadoop.fs.FileSystem;
>>> import org.apache.hadoop.fs.FSDataInputStream;
>>> import org.apache.hadoop.fs.FSDataOutputStream;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IOUtils;
>>>
>>> public class HadoopDFSFileReadWrite {
>>>
>>> static void usage () {
>>> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>>> <output file>");
>>> System.exit(1);
>>> }
>>>
>>> static void printAndExit(String str) {
>>> System.err.println(str);
>>> System.exit(1);
>>> }
>>>
>>> public static void main (String[] argv) throws IOException {
>>> Configuration conf = new Configuration();
>>> conf.set("fs.default.name", "localhost:9000");
>>> FileSystem fs = FileSystem.get(conf);
>>>
>>> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>> for(FileStatus status : fileStatus) {
>>> System.out.println("File: " + status.getPath());
>>> }
>>>
>>> if (argv.length != 2)
>>> usage();
>>>
>>> // HadoopDFS deals with Path
>>> Path inFile = new Path(argv[0]);
>>> Path outFile = new Path(argv[1]);
>>>
>>> // Check if input/output are valid
>>> if (!fs.exists(inFile))
>>> printAndExit("Input file not found");
>>> if (!fs.isFile(inFile))
>>> printAndExit("Input should be a file");
>>> if (fs.exists(outFile))
>>> printAndExit("Output already exists");
>>>
>>> // Read from and write to new file
>>> FSDataInputStream in = fs.open(inFile);
>>> FSDataOutputStream out = fs.create(outFile);
>>> byte buffer[] = new byte[256];
>>> try {
>>> int bytesRead = 0;
>>> while ((bytesRead = in.read(buffer)) > 0) {
>>> out.write(buffer, 0, bytesRead);
>>> }
>>>
>>> } catch (IOException e) {
>>> System.out.println("Error while copying file");
>>> } finally {
>>> in.close();
>>> out.close();
>>> }
>>> }
>>> }
>>>
>>> Suresh
>>>
>>>
>>> -----Original Message-----
>>> From: Ajey Shah [mailto:[EMAIL PROTECTED]
>>> Sent: Thursday, May 01, 2008 3:31 AM
>>> To: [email protected]
>>> Subject: How do I copy files from my linux file system to HDFS using a
>>> java prog?
>>>
>>>
>>> Hello all,
>>>
>>> I need to copy files from my linux file system to HDFS in a java program
>>> and not manually. This is the piece of code that I have.
>>>
>>> try {
>>>
>>> FileSystem hdfs = FileSystem.get(new
>>> Configuration());
>>>
>>> LocalFileSystem ls = null;
>>>
>>> ls = hdfs.getLocal(hdfs.getConf());
>>>
>>> hdfs.copyFromLocalFile(false, new
>>> Path(fileName), new Path(outputFile));
>>>
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> }
>>>
>>> The problem is that it searches for the input path on the HDFS and not
>>> my linux file system.
>>>
>>> Can someone point out where I may be wrong. I feel it's some
>>> configuration issue but have not been able to figure it out.
>>>
>>> Thanks.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>>> DFS-using-a-java-prog--tp16992491p16992491.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>>
>>>
>>>
>
>
>
--
View this message in context:
http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136713.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
