Maha, If you want to access hdfs file in mapper class, use context.getConfiguration() in FileSystem.get. In the main method, you need to create configuration first using new configuration(). In your case, you cannot create configuration in main method, and access it in mapper class because the main method and the mapper are not in the same jvm.
Wei -----Original Message----- From: Hari Sreekumar [mailto:[email protected]] Sent: Sunday, December 19, 2010 8:52 AM To: [email protected] Subject: Re: Needs a simple answer You should not be creating new configuration instances in the mapper/reducer. Instead, get the job configuration using context.getConfiguration() [API 0.20.2] On Sun, Dec 19, 2010 at 1:16 PM, maha <[email protected]> wrote: > Thanks Ping :) actually my mistake was that I used to create a FileSystem > in the Mapper class as: > > fs = FileSystem.get( myConf) ; // where myConf is defined in the > main class WordCount. > > Somehow, that wasn't working .. but as soon as I changed it to be with new > Configurations: > > FileSystem fs = FileSystem.get(new Configuration()) // it worked ! > > > Any reason for that? > > Thank you, > Maha > > > > On Dec 17, 2010, at 2:59 PM, Peng, Wei wrote: > > > > > You can put your local file to distributed file system by hadoop fs -put > localfile DFSfile. > > Then access it by > > Configuration conf = new Configuration(); > > try { > > FileSystem fs = FileSystem.get(URI.create(DFSfile), > conf); > > FSDataInputStream in = null; > > in = fs.open(new Path(path)); > > String content = in.readUTF(); > > > > } catch (IOException e) { > > > > } > > -----Original Message----- > > From: maha [mailto:[email protected]] > > Sent: Fri 12/17/2010 2:20 PM > > To: [email protected] > > Subject: Re: Needs a simple answer > > > > Hi Ted, thanks for the hint , but I'm using only one machine now .. so > the file is definitely in there. > > > > So, How can I open an hdfs- file in my InputFormat class (in GetSplits) > so I can pass the contents to the mappers ? > > > > Now, the splits are splits of files-paths from the input directory ( eg. > split1: /tmp/f1, split2:/tmp/f2 .... split4: /tmp/f4) instead I want -----> > ( split1: content of file1 , ....). > > > > Thank you, > > > > Maha > > > > > > On Dec 16, 2010, at 2:49 PM, Ted Dunning wrote: > > > >> Maha, > >> > >> Remember that the mapper is not running on the same machine as the main > >> class. Thus local files aren't where you think. > >> > >> On Thu, Dec 16, 2010 at 1:06 PM, maha <[email protected]> wrote: > >> > >>> Hi all, > >>> > >>> Why the following lines would work in the main class (WordCount) and > not > >>> in Mapper ? even though " myconf " is set in WordCount to point to the > >>> getConf() returned object. > >>> > >>> try{ > >>> FileSystem hdfs = FileSystem.get(wc.WordCount.myconf); > >>> hdfs.copyFromLocalFile(new Path("/Users/file"), new > >>> Path("/tmp/file")); > >>> }catch(Exception e) { System.err.print("\nError");} > >>> > >>> > >>> Also, the print statement will never print on console unless it's in my > >>> run function.. > >>> > >>> Appreciate it :) > >>> > >>> Maha > >>> > >>> > > > > > >
