> On March 6, 2014, 8:04 a.m., Rohini Palaniswamy wrote: > > src/org/apache/pig/pen/IllustratorAttacher.java, lines 418-421 > > <https://reviews.apache.org/r/15881/diff/3/?file=509987#file509987line418> > > > > Don't we need the equivalent of this code in visitPackage?
The Packager is now responsible for this. It is present in JoinPackager. - Mark ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15881/#review36273 ----------------------------------------------------------- On April 3, 2014, 2:57 a.m., Mark Wagner wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15881/ > ----------------------------------------------------------- > > (Updated April 3, 2014, 2:57 a.m.) > > > Review request for pig and Cheolsoo Park. > > > Bugs: PIG-3591 > https://issues.apache.org/jira/browse/PIG-3591 > > > Repository: pig-git > > > Description > ------- > > Separate "packaging" logic from "shuffle handling" logic. This moves the > packaging logic to a new class "Packager", which is extended by > CombinePackager, LitePackager, MultiQueryPackager, and JoinPackager. > > This is not finished. Known problem are illustrate and streaming the last > input are not implemented. > > > Diffs > ----- > > src/org/apache/pig/backend/hadoop/executionengine/fetch/FetchOptimizer.java > d801f6f > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java > 3638b5c > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/CombinerOptimizer.java > 18a382b > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 5e28eb6 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java > 5dddab7 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java > 93de6d5 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java > eb7c428 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java > 64f0ee1 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java > 933363d > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java > 773a22c > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java > eea5ce3 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java > 1578630 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java > 47137d5 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java > abb16ff > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java > ff82801 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java > 892c26f > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java > 9105a0e > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java > 82f11ac > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java > d604174 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java > 86314d9 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java > c200715 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java > a92e643 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java > b860521 > src/org/apache/pig/data/ReadOnceBag.java e2b3887 > src/org/apache/pig/data/SelfSpillBag.java 2fcf36a > > src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java > 7112695 > src/org/apache/pig/pen/IllustratorAttacher.java db9c707 > src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7 > test/org/apache/pig/test/TestExampleGenerator.java 42f757f > test/org/apache/pig/test/TestJobSubmission.java cccade4 > test/org/apache/pig/test/TestPackage.java 6b197d1 > test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a > test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728 > test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f > test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d > test/org/apache/pig/test/data/GoldenFiles/MRC12.gld 15f1158 > test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b > test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509 > test/org/apache/pig/test/data/GoldenFiles/MRC15.gld e8bd281 > test/org/apache/pig/test/data/GoldenFiles/MRC16.gld 6151d1e > test/org/apache/pig/test/data/GoldenFiles/MRC17.gld cec5d57 > test/org/apache/pig/test/data/GoldenFiles/MRC18.gld 01cdc32 > test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 8688178 > test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0 > test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f > test/org/apache/pig/test/data/GoldenFiles/MRC4.gld 7c5078e > test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 45cba2d > test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 09fcaec > test/org/apache/pig/test/data/GoldenFiles/MRC8.gld 2d301a9 > test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c5d7047 > > Diff: https://reviews.apache.org/r/15881/diff/ > > > Testing > ------- > > ant test-commit passes, with the exception of TestExampleGenerator which is > caused by the illustrate work not being finished. I also ran some manual > scripts, which worked. > > > Thanks, > > Mark Wagner > >
