You can put your local file to distributed file system by hadoop fs -put
localfile DFSfile.
Then access it by
Configuration conf = new Configuration();
try {
FileSystem fs = FileSystem.get(URI.create(DFSfile),
conf);
FSDataInputStream in = null;
in = fs.open(new Path(path));
String content = in.readUTF();
} catch (IOException e) {
}
-----Original Message-----
From: maha [mailto:[email protected]]
Sent: Fri 12/17/2010 2:20 PM
To: [email protected]
Subject: Re: Needs a simple answer
Hi Ted, thanks for the hint , but I'm using only one machine now .. so the file
is definitely in there.
So, How can I open an hdfs- file in my InputFormat class (in GetSplits) so I
can pass the contents to the mappers ?
Now, the splits are splits of files-paths from the input directory ( eg.
split1: /tmp/f1, split2:/tmp/f2 .... split4: /tmp/f4) instead I want -----> (
split1: content of file1 , ....).
Thank you,
Maha
On Dec 16, 2010, at 2:49 PM, Ted Dunning wrote:
> Maha,
>
> Remember that the mapper is not running on the same machine as the main
> class. Thus local files aren't where you think.
>
> On Thu, Dec 16, 2010 at 1:06 PM, maha <[email protected]> wrote:
>
>> Hi all,
>>
>> Why the following lines would work in the main class (WordCount) and not
>> in Mapper ? even though " myconf " is set in WordCount to point to the
>> getConf() returned object.
>>
>> try{
>> FileSystem hdfs = FileSystem.get(wc.WordCount.myconf);
>> hdfs.copyFromLocalFile(new Path("/Users/file"), new
>> Path("/tmp/file"));
>> }catch(Exception e) { System.err.print("\nError");}
>>
>>
>> Also, the print statement will never print on console unless it's in my
>> run function..
>>
>> Appreciate it :)
>>
>> Maha
>>
>>
