[ http://issues.apache.org/jira/browse/HADOOP-723?page=comments#action_12450178 ] eric baldeschwieler commented on HADOOP-723: --------------------------------------------
You might want to use a temp file and rename strategy to make sure: a) you only delete files you own b) it is clear that a file was completely transfered by inspection. c) temp files can be easily cleaned up in some conditions (restart say) > Race condition exists in the method MapOutputLocation.getFile > ------------------------------------------------------------- > > Key: HADOOP-723 > URL: http://issues.apache.org/jira/browse/HADOOP-723 > Project: Hadoop > Issue Type: Bug > Components: mapred > Reporter: Devaraj Das > > There seems to be a race condition in the way the Reduces copy the map output > files from the Maps. If a copier is blocked in the connect method (in the > beginning of the method MapOutputLocation.getFile) to a Jetty on a Map, and > the MapCopyLeaseChecker detects that the copier was idle for too long, it > will go ahead and issue a interrupt (read 'kill') to this thread and create a > new Copier thread. However, the copier, currently blocked trying to connect > to Jetty on a Map, doesn't actually get killed until the connect timeout > expires and as soon as the connect comes out (with an IOException), it will > delete the map output file which actually could have been (successfully) > created by the new Copier thread. This leads to the Sort phase for that > reducer failing with a FileNotFoundException. > One simple way to fix this is to not delete the file if the file was not > created within this getFile method. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
