Author: pradeepkth
Date: Tue Apr 20 16:24:34 2010
New Revision: 935981
URL: http://svn.apache.org/viewvc?rev=935981&view=rev
Log:
PIG-1290: WeightedRangePartitioner should not check if input is empty if
quantile file is empty (pradeepkth)
Modified:
hadoop/pig/branches/branch-0.6/CHANGES.txt
hadoop/pig/branches/branch-0.6/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
Modified: hadoop/pig/branches/branch-0.6/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/pig/branches/branch-0.6/CHANGES.txt?rev=935981&r1=935980&r2=935981&view=diff
==============================================================================
--- hadoop/pig/branches/branch-0.6/CHANGES.txt (original)
+++ hadoop/pig/branches/branch-0.6/CHANGES.txt Tue Apr 20 16:24:34 2010
@@ -156,6 +156,9 @@ PIG-922: Logical optimizer: push up proj
BUG FIXES
+PIG-1290: WeightedRangePartitioner should not check if input is empty if
+quantile file is empty (pradeepkth)
+
PIG-1364: Public javadoc on apache site still on 0.2, needs to be updated for
each version release (gates)
PIG-1346: In unit tests Util.executeShellCommand relies on java commands
Modified:
hadoop/pig/branches/branch-0.6/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
URL:
http://svn.apache.org/viewvc/hadoop/pig/branches/branch-0.6/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java?rev=935981&r1=935980&r2=935981&view=diff
==============================================================================
---
hadoop/pig/branches/branch-0.6/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
(original)
+++
hadoop/pig/branches/branch-0.6/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java
Tue Apr 20 16:24:34 2010
@@ -103,17 +103,12 @@ public class WeightedRangePartitioner im
new DiscreteProbabilitySampleGenerator(probVec));
}
}
- else
- {
- ArrayList<Pair<FileSpec, Boolean>> inp =
(ArrayList<Pair<FileSpec,
Boolean>>)ObjectSerializer.deserialize(job.get("pig.inputs", ""));
- String inputFileName = inp.get(0).first.getFileName();
- long inputSize = FileLocalizer.getSize(inputFileName);
- if (inputSize!=0)
- {
- throw new RuntimeException("Empty samples file and
non-empty input file");
- }
- // Otherwise, we do not put anything to weightedParts
- }
+ // else - the quantiles file is empty - unless we have a bug, the
+ // input must also be empty in which case we don't need to put
+ // anything in weightedParts since getPartition() should never get
+ // called. If the quantiles file is empty due to either a bug or
+ // a transient failure situation on the dfs, then weightedParts
will
+ // not be populated and the job will fail in getPartition()
}catch (Exception e){
throw new RuntimeException(e);
}
