You should not be creating new configuration instances in the mapper/reducer. Instead, get the job configuration using context.getConfiguration() [API 0.20.2]
On Sun, Dec 19, 2010 at 1:16 PM, maha <[email protected]> wrote: > Thanks Ping :) actually my mistake was that I used to create a FileSystem > in the Mapper class as: > > fs = FileSystem.get( myConf) ; // where myConf is defined in the > main class WordCount. > > Somehow, that wasn't working .. but as soon as I changed it to be with new > Configurations: > > FileSystem fs = FileSystem.get(new Configuration()) // it worked ! > > > Any reason for that? > > Thank you, > Maha > > > > On Dec 17, 2010, at 2:59 PM, Peng, Wei wrote: > > > > > You can put your local file to distributed file system by hadoop fs -put > localfile DFSfile. > > Then access it by > > Configuration conf = new Configuration(); > > try { > > FileSystem fs = FileSystem.get(URI.create(DFSfile), > conf); > > FSDataInputStream in = null; > > in = fs.open(new Path(path)); > > String content = in.readUTF(); > > > > } catch (IOException e) { > > > > } > > -----Original Message----- > > From: maha [mailto:[email protected]] > > Sent: Fri 12/17/2010 2:20 PM > > To: [email protected] > > Subject: Re: Needs a simple answer > > > > Hi Ted, thanks for the hint , but I'm using only one machine now .. so > the file is definitely in there. > > > > So, How can I open an hdfs- file in my InputFormat class (in GetSplits) > so I can pass the contents to the mappers ? > > > > Now, the splits are splits of files-paths from the input directory ( eg. > split1: /tmp/f1, split2:/tmp/f2 .... split4: /tmp/f4) instead I want -----> > ( split1: content of file1 , ....). > > > > Thank you, > > > > Maha > > > > > > On Dec 16, 2010, at 2:49 PM, Ted Dunning wrote: > > > >> Maha, > >> > >> Remember that the mapper is not running on the same machine as the main > >> class. Thus local files aren't where you think. > >> > >> On Thu, Dec 16, 2010 at 1:06 PM, maha <[email protected]> wrote: > >> > >>> Hi all, > >>> > >>> Why the following lines would work in the main class (WordCount) and > not > >>> in Mapper ? even though " myconf " is set in WordCount to point to the > >>> getConf() returned object. > >>> > >>> try{ > >>> FileSystem hdfs = FileSystem.get(wc.WordCount.myconf); > >>> hdfs.copyFromLocalFile(new Path("/Users/file"), new > >>> Path("/tmp/file")); > >>> }catch(Exception e) { System.err.print("\nError");} > >>> > >>> > >>> Also, the print statement will never print on console unless it's in my > >>> run function.. > >>> > >>> Appreciate it :) > >>> > >>> Maha > >>> > >>> > > > > > >
