Re: Review Request 15881: PIG-3591: Refactor POPackage

Mark Wagner Fri, 04 Apr 2014 19:26:30 -0700

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15881/
-----------------------------------------------------------


(Updated April 5, 2014, 2:25 a.m.)


Review request for pig and Cheolsoo Park.


Bugs: PIG-3591
    https://issues.apache.org/jira/browse/PIG-3591


Repository: pig-git


Description
-------

Separate "packaging" logic from "shuffle handling" logic. This moves the 
packaging logic to a new class "Packager", which is extended by 
CombinePackager, LitePackager, MultiQueryPackager, and JoinPackager.

This is not finished. Known problem are illustrate and streaming the last input 
are not implemented.


Diffs (updated)
-----

  src/org/apache/pig/backend/hadoop/executionengine/fetch/FetchOptimizer.java 
d801f6f 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java
 3638b5c 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/CombinerOptimizer.java
 18a382b 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
 b7ce7e2 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java
 5dddab7 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 
93de6d5 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
 d3ebeb3 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java
 64f0ee1 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java
 933363d 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java
 773a22c 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
 eea5ce3 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java
 54740a0 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java
 47137d5 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java
 abb16ff 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java
 ff82801 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java
 892c26f 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java
 9105a0e 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java
 82f11ac 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java
 d604174 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java
 86314d9 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java
 c200715 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java
 a92e643 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java
 b860521 
  src/org/apache/pig/data/ReadOnceBag.java e2b3887 
  src/org/apache/pig/data/SelfSpillBag.java 2fcf36a 
  src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 
7112695 
  src/org/apache/pig/pen/IllustratorAttacher.java db9c707 
  src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7 
  test/org/apache/pig/test/TestExampleGenerator.java 42f757f 
  test/org/apache/pig/test/TestJobSubmission.java 9741dc9 
  test/org/apache/pig/test/TestPackage.java 6b197d1 
  test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a 
  test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728 
  test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f 
  test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d 
  test/org/apache/pig/test/data/GoldenFiles/MRC12.gld 15f1158 
  test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b 
  test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509 
  test/org/apache/pig/test/data/GoldenFiles/MRC15.gld e8bd281 
  test/org/apache/pig/test/data/GoldenFiles/MRC16.gld 6151d1e 
  test/org/apache/pig/test/data/GoldenFiles/MRC17.gld cec5d57 
  test/org/apache/pig/test/data/GoldenFiles/MRC18.gld 01cdc32 
  test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 8688178 
  test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0 
  test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f 
  test/org/apache/pig/test/data/GoldenFiles/MRC4.gld 7c5078e 
  test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 45cba2d 
  test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 09fcaec 
  test/org/apache/pig/test/data/GoldenFiles/MRC8.gld 2d301a9 
  test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c5d7047 

Diff: https://reviews.apache.org/r/15881/diff/


Testing
-------

ant test-commit passes, with the exception of TestExampleGenerator which is 
caused by the illustrate work not being finished. I also ran some manual 
scripts, which worked.


Thanks,

Mark Wagner

Re: Review Request 15881: PIG-3591: Refactor POPackage

Reply via email to