----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27429/ -----------------------------------------------------------
(Updated Oct. 31, 2014, 11:10 p.m.)
Review request for pig and Daniel Dai.
Changes
-------
Fixed test failures -
TestTezAutoParallelism.testSkewedJoinIncreaseIntermediateParallelism() and
TestTezCompiler.testUnionLimit(). Also reduced memory used in unit test from 2G
to 512M. That is good enough and also makes the tests takes 10-20 mins less on
test-tez compared to last few times I ran it.
Bugs: PIG-4259
https://issues.apache.org/jira/browse/PIG-4259
Repository: pig
Description
-------
Patch addresses different issues encountered while trying to debug wrong
results for a production script.
Issues addressed:
- Optimized union followed directly by Limit also fixing possibility of
incorrect results when Limit could be totally removed by UnionOptimizer if
parallelism of union was also 1.
- Fixed wrong result in case of group by with secondary key followed by
Union (Union_14)
- Fixed CROSS for Union and multiquery.
- Fixed/Optimized POLimit to not process next input in bag redundantly if
limit is already reached.
- Fixed some issues in auto parallelism and modified overriding
parallelism of intermediate reducers (PIG-4162) only for required cases.
- Adjust the AM size based on total tasks. Pain to keep adjusting memory
size after task runs for a long time and then fails with OOM.
- Fixes NPE in logs while fetching counters when job fails
- Avoid printing counters everytime while printing dagStatus. Only print
tasks and diagnostics.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/pig/trunk/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLimit.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobCompiler.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezCompiler.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperPlan.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/CombinerOptimizer.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/MultiQueryOptimizerTez.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/ParallelismSetter.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/TezOperDependencyParallelismEstimator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/UnionOptimizer.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/builtin/GFCross.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/Utils.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/tests/nightly.conf
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/Util.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/tez/TEZC-Union-8-OPTOFF.gld
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/tez/TEZC-Union-8.gld
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/tez/TestTezAutoParallelism.java
1635795
Diff: https://reviews.apache.org/r/27429/diff/
Testing
-------
Added unit and e2e tests. In the process of running full suite of unit and e2e
tests.
Thanks,
Rohini Palaniswamy
