Author: ddas
Date: Wed Dec 26 00:37:27 2007
New Revision: 606854
URL: http://svn.apache.org/viewvc?rev=606854&view=rev
Log:
HADOOP-2486. When the list of files from the InMemoryFileSystem is obtained for
merging, this patch will ensure that only those files whose checksums have also
got created (renamed) are returned. Contributed by Devaraj Das.
Modified:
lucene/hadoop/trunk/CHANGES.txt
lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java
Modified: lucene/hadoop/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=606854&r1=606853&r2=606854&view=diff
==============================================================================
--- lucene/hadoop/trunk/CHANGES.txt (original)
+++ lucene/hadoop/trunk/CHANGES.txt Wed Dec 26 00:37:27 2007
@@ -333,6 +333,10 @@
round-robin disk selections randomly. This helps in spreading data across
multiple partitions much better. (acmurhty)
+ HADOOP-2486. When the list of files from the InMemoryFileSystem is obtained
+ for merging, this patch will ensure that only those files whose checksums
+ have also got created (renamed) are returned. (ddas)
+
IMPROVEMENTS
HADOOP-2160. Remove project-level, non-user documentation from
Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java
URL:
http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java?rev=606854&r1=606853&r2=606854&view=diff
==============================================================================
--- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java
(original)
+++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java Wed
Dec 26 00:37:27 2007
@@ -1386,8 +1386,11 @@
public void run() {
LOG.info(reduceTask.getTaskId() + " Thread started: " + getName());
try {
+ Path[] inMemClosedFiles;
//initiate merge
- Path[] inMemClosedFiles = inMemFileSys.getFiles(MAP_OUTPUT_FILTER);
+ synchronized (ReduceTask.this) {
+ inMemClosedFiles = inMemFileSys.getFiles(MAP_OUTPUT_FILTER);
+ }
//Note that the above Path[] could be of length 0 if all copies are
//in flight. So we make sure that we have some 'closed' map
//output files to merge to get the benefit of in-memory merge
