Thanks Ping :) actually my mistake was that I used to create a FileSystem in
the Mapper class as:
fs = FileSystem.get( myConf) ; // where myConf is defined in the main
class WordCount.
Somehow, that wasn't working .. but as soon as I changed it to be with new
Configurations:
FileSystem fs = FileSystem.get(new Configuration()) // it worked !
Any reason for that?
Thank you,
Maha
On Dec 17, 2010, at 2:59 PM, Peng, Wei wrote:
>
> You can put your local file to distributed file system by hadoop fs -put
> localfile DFSfile.
> Then access it by
> Configuration conf = new Configuration();
> try {
> FileSystem fs = FileSystem.get(URI.create(DFSfile),
> conf);
> FSDataInputStream in = null;
> in = fs.open(new Path(path));
> String content = in.readUTF();
>
> } catch (IOException e) {
>
> }
> -----Original Message-----
> From: maha [mailto:[email protected]]
> Sent: Fri 12/17/2010 2:20 PM
> To: [email protected]
> Subject: Re: Needs a simple answer
>
> Hi Ted, thanks for the hint , but I'm using only one machine now .. so the
> file is definitely in there.
>
> So, How can I open an hdfs- file in my InputFormat class (in GetSplits) so I
> can pass the contents to the mappers ?
>
> Now, the splits are splits of files-paths from the input directory ( eg.
> split1: /tmp/f1, split2:/tmp/f2 .... split4: /tmp/f4) instead I want ----->
> ( split1: content of file1 , ....).
>
> Thank you,
>
> Maha
>
>
> On Dec 16, 2010, at 2:49 PM, Ted Dunning wrote:
>
>> Maha,
>>
>> Remember that the mapper is not running on the same machine as the main
>> class. Thus local files aren't where you think.
>>
>> On Thu, Dec 16, 2010 at 1:06 PM, maha <[email protected]> wrote:
>>
>>> Hi all,
>>>
>>> Why the following lines would work in the main class (WordCount) and not
>>> in Mapper ? even though " myconf " is set in WordCount to point to the
>>> getConf() returned object.
>>>
>>> try{
>>> FileSystem hdfs = FileSystem.get(wc.WordCount.myconf);
>>> hdfs.copyFromLocalFile(new Path("/Users/file"), new
>>> Path("/tmp/file"));
>>> }catch(Exception e) { System.err.print("\nError");}
>>>
>>>
>>> Also, the print statement will never print on console unless it's in my
>>> run function..
>>>
>>> Appreciate it :)
>>>
>>> Maha
>>>
>>>
>
>
