Thanks Suresh. It works! :)
Babu, Suresh wrote: > > This program can read from local file system as well as HDFS. > Try <java program> <Localfile> <HDFSDestination> > > Thanks > Suresh > > -----Original Message----- > From: Ajey Shah [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 07, 2008 4:04 AM > To: [email protected] > Subject: RE: How do I copy files from my linux file system to HDFS using > a java prog? > > > Thanks Suresh. But even this program reads and writes from the HDFS. > What i need to do is read from my normal local linux harddrive and write > to the HDFS. > > I'm sorry if I misunderstood your program. > > Thanks for replying. :) > > > > Babu, Suresh wrote: >> >> >> Try this program. Modify the HDFS configuration, if it is different >> from the default. >> >> import java.io.File; >> import java.io.IOException; >> >> import org.apache.hadoop.conf.Configuration; >> import org.apache.hadoop.fs.FileStatus; import >> org.apache.hadoop.fs.FileSystem; import >> org.apache.hadoop.fs.FSDataInputStream; >> import org.apache.hadoop.fs.FSDataOutputStream; >> import org.apache.hadoop.fs.Path; >> import org.apache.hadoop.io.IOUtils; >> >> public class HadoopDFSFileReadWrite { >> >> static void usage () { >> System.out.println("Usage : HadoopDFSFileReadWrite <inputfile> >> <output file>"); >> System.exit(1); >> } >> >> static void printAndExit(String str) { >> System.err.println(str); >> System.exit(1); >> } >> >> public static void main (String[] argv) throws IOException { >> Configuration conf = new Configuration(); >> conf.set("fs.default.name", "localhost:9000"); >> FileSystem fs = FileSystem.get(conf); >> >> FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory()); >> for(FileStatus status : fileStatus) { >> System.out.println("File: " + status.getPath()); >> } >> >> if (argv.length != 2) >> usage(); >> >> // HadoopDFS deals with Path >> Path inFile = new Path(argv[0]); >> Path outFile = new Path(argv[1]); >> >> // Check if input/output are valid >> if (!fs.exists(inFile)) >> printAndExit("Input file not found"); >> if (!fs.isFile(inFile)) >> printAndExit("Input should be a file"); >> if (fs.exists(outFile)) >> printAndExit("Output already exists"); >> >> // Read from and write to new file >> FSDataInputStream in = fs.open(inFile); >> FSDataOutputStream out = fs.create(outFile); >> byte buffer[] = new byte[256]; >> try { >> int bytesRead = 0; >> while ((bytesRead = in.read(buffer)) > 0) { >> out.write(buffer, 0, bytesRead); >> } >> >> } catch (IOException e) { >> System.out.println("Error while copying file"); >> } finally { >> in.close(); >> out.close(); >> } >> } >> } >> >> Suresh >> >> >> -----Original Message----- >> From: Ajey Shah [mailto:[EMAIL PROTECTED] >> Sent: Thursday, May 01, 2008 3:31 AM >> To: [email protected] >> Subject: How do I copy files from my linux file system to HDFS using a > >> java prog? >> >> >> Hello all, >> >> I need to copy files from my linux file system to HDFS in a java >> program and not manually. This is the piece of code that I have. >> >> try { >> >> FileSystem hdfs = FileSystem.get(new > Configuration()); >> >> LocalFileSystem ls = null; >> >> ls = hdfs.getLocal(hdfs.getConf()); >> >> hdfs.copyFromLocalFile(false, new >> Path(fileName), new Path(outputFile)); >> >> } catch (Exception e) { >> e.printStackTrace(); >> } >> >> The problem is that it searches for the input path on the HDFS and not > >> my linux file system. >> >> Can someone point out where I may be wrong. I feel it's some >> configuration issue but have not been able to figure it out. >> >> Thanks. >> -- >> View this message in context: >> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to >> -H DFS-using-a-java-prog--tp16992491p16992491.html >> Sent from the Hadoop core-user mailing list archive at Nabble.com. >> >> >> > > -- > View this message in context: > http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H > DFS-using-a-java-prog--tp16992491p17093646.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. > > > -- View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136728.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
