svn commit: r512936 - in /lucene/hadoop/trunk: CHANGES.txt src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java

[cutting]

Author: cutting
Date: Wed Feb 28 11:59:40 2007
New Revision: 512936

URL: http://svn.apache.org/viewvc?view=rev&rev=512936
Log:
HADOOP-1043.  Optimize shuffle, increasing parallelism.  Contributed by Devaraj.

Modified:
    lucene/hadoop/trunk/CHANGES.txt
    lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java

Modified: lucene/hadoop/trunk/CHANGES.txt
URL: 
http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?view=diff&rev=512936&r1=512935&r2=512936
==============================================================================
--- lucene/hadoop/trunk/CHANGES.txt (original)
+++ lucene/hadoop/trunk/CHANGES.txt Wed Feb 28 11:59:40 2007
@@ -159,6 +159,9 @@
 47. HADOOP-972.  Optimize HDFS's rack-aware block placement algorithm.
     (Hairong Kuang via cutting)
 
+48. HADOOP-1043.  Optimize shuffle, increasing parallelism.
+    (Devaraj Das via cutting)
+
 
 Release 0.11.2 - 2007-02-16
 

Modified: 
lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java
URL: 
http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java?view=diff&rev=512936&r1=512935&r2=512936
==============================================================================
--- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java 
(original)
+++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskRunner.java 
Wed Feb 28 11:59:40 2007
@@ -608,9 +608,17 @@
           numInFlight--;
         }
         
+        boolean busy = true;
         // ensure we have enough to keep us busy
         if (numInFlight < lowThreshold && (numOutputs-numCopied) > 
probe_sample_size) {
-          break;
+          busy = false;
+        }
+        //Check whether we have more CopyResult to check. If there is none, and
+        //we are not busy enough, break
+        synchronized (copyResults) {
+          if (copyResults.size() == 0 && !busy) {
+            break;
+          }
         }
       }