Look like you use your reducer class as the combiner. The combiner will be called from mappers, potentially for multiple times. If you want to create side files in reducer, you cannot use that class as the combiner.
Runping > -----Original Message----- > From: Zhang, jian [mailto:[EMAIL PROTECTED] > Sent: Thursday, April 10, 2008 11:17 PM > To: core-user@hadoop.apache.org > Subject: What's the proper way to use hadoop task side-effect files? > > Hi, > > I was new to hadoop. Sorry for my novice question. > I got some problem while I was trying to use task side-effect files. > Since there is no code example in wiki, I tried this way: > > I override cofigure method in reducer to create a side file, > > public void configure(JobConf conf){ > logger.info("Tring to create sideFiles inside reducer.!"); > > Path workpath=conf.getOutputPath(); > Path sideFile= new Path(workpath,"SideFile.txt"); > try { > FileSystem fs = FileSystem.get(conf); > out= fs.create(sideFile); > } catch (IOException e) { > logger.error("Failed to create sidefile!"); > } > } > And try to use it in reducer. > > But I got some strange problems, > Even If the method is in reducer Class, mapper tasks are creating the > side files. > Mapper tasks hang because there are tring to recreate the file. > > org.apache.hadoop.dfs.AlreadyBeingCreatedException: > failed to create file > /data/input/MID06/_temporary/_task_200804112315_0001_m_000008_0/SideFile > .txt for DFSClient_task_200804112315_0001_m_000008_0 on client > 192.168.0.203 because current leaseholder is trying to recreate file. > at > org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:9 > 74) > at > org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:931) > at org.apache.hadoop.dfs.NameNode.create(NameNode.java:281) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav > a:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:899) > > > Can anybody help me on this, how to use side-effect files? > > > > Best Regards > > Jian Zhang
