----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27429/#review59442 -----------------------------------------------------------
Ship it! Ship It! - Daniel Dai On Oct. 31, 2014, 11:10 p.m., Rohini Palaniswamy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27429/ > ----------------------------------------------------------- > > (Updated Oct. 31, 2014, 11:10 p.m.) > > > Review request for pig and Daniel Dai. > > > Bugs: PIG-4259 > https://issues.apache.org/jira/browse/PIG-4259 > > > Repository: pig > > > Description > ------- > > Patch addresses different issues encountered while trying to debug wrong > results for a production script. > > Issues addressed: > - Optimized union followed directly by Limit also fixing possibility of > incorrect results when Limit could be totally removed by UnionOptimizer if > parallelism of union was also 1. > - Fixed wrong result in case of group by with secondary key followed by > Union (Union_14) > - Fixed CROSS for Union and multiquery. > - Fixed/Optimized POLimit to not process next input in bag redundantly > if limit is already reached. > - Fixed some issues in auto parallelism and modified overriding > parallelism of intermediate reducers (PIG-4162) only for required cases. > - Adjust the AM size based on total tasks. Pain to keep adjusting memory > size after task runs for a long time and then fails with OOM. > - Fixes NPE in logs while fetching counters when job fails > - Avoid printing counters everytime while printing dagStatus. Only print > tasks and diagnostics. > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/pig/trunk/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLimit.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobCompiler.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezCompiler.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperPlan.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperator.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/CombinerOptimizer.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/MultiQueryOptimizerTez.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/ParallelismSetter.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/TezOperDependencyParallelismEstimator.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/UnionOptimizer.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/builtin/GFCross.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/Utils.java > 1635795 > http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/tests/nightly.conf > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/Util.java > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/tez/TEZC-Union-8-OPTOFF.gld > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/tez/TEZC-Union-8.gld > 1635795 > > http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/tez/TestTezAutoParallelism.java > 1635795 > > Diff: https://reviews.apache.org/r/27429/diff/ > > > Testing > ------- > > Added unit and e2e tests. In the process of running full suite of unit and > e2e tests. > > > Thanks, > > Rohini Palaniswamy > >
