Check out Pages 217-220 of the Hadoop: The Definitive Guide book. It has some nice explanation. Also, http://whiteycode.blogspot.com/2012/06/hadoop-removing-empty-output-files.html
Plus as Jay said, explanation of your use-case might also be helpful. Regards, Shahab On Mon, Jun 3, 2013 at 5:44 PM, samir das mohapatra <[email protected] > wrote: > Do you have any link or example ? could you please send me ? > > > On Tue, Jun 4, 2013 at 2:53 AM, Shahab Yunus <[email protected]>wrote: > >> Have you taken a look into extending the FileOutputFormat class and >> overriding the OutputCommitter API functionality? >> >> Regards, >> Shahab >> >> >> On Mon, Jun 3, 2013 at 5:11 PM, samir das mohapatra < >> [email protected]> wrote: >> >>> Dear All, >>> >>> Is there any way to copy the intermediate output file of the mapper >>> into local folder after each map task complete. >>> >>> Right now I am using >>> >>> FileSystem.copyToLocalFile(hdfsLocation,localLocation); >>> indiste the cleanup of mapper task , but it is failing . >>> >>> Exception file not found. >>> >>> But if I am giving same statement after the job complete in driver class >>> ,it is working fine. that i dont want. >>> >>> protected void cleanup(Context context){ >>> FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed >>> } >>> >>> Note: I need to copy the inter mediate output of the mapper to local >>> file system just after each map task complete. I dont want any reducer. >>> >>> If this is not the right solution then how to solve this type of >>> scenario. >>> >>> Any help............. >>> >>> regards, >>> samir. >>> >> >> >
