Review Request 15881: PIG-3591: Refactor POPackage

Mark Wagner Wed, 27 Nov 2013 10:05:54 -0800

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15881/
-----------------------------------------------------------


Review request for pig and Cheolsoo Park.


Bugs: PIG-3591
    https://issues.apache.org/jira/browse/PIG-3591


Repository: pig-git


Description
-------

Separate "packaging" logic from "shuffle handling" logic. This moves the 
packaging logic to a new class "Packager", which is extended by 
CombinePackager, LitePackager, MultiQueryPackager, and JoinPackager.

This is not finished. Known problem are illustrate and streaming the last input 
are not implemented.


Diffs
-----

  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java
 3638b5c 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/CombinerOptimizer.java
 18a382b 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
 fc9261c 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java
 9341153 
  src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java 
93de6d5 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
 9580c26 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java
 64f0ee1 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java
 933363d 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java
 906930a 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
 eea5ce3 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java
 1578630 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java
 47137d5 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java
 abb16ff 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java
 ff82801 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java
 892c26f 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java
 9105a0e 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java
 82f11ac 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java
 d604174 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java
 86314d9 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java
 c200715 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java
 b860521 
  src/org/apache/pig/data/ReadOnceBag.java e2b3887 
  src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java 
7112695 
  src/org/apache/pig/pen/IllustratorAttacher.java db9c707 
  src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7 
  test/org/apache/pig/test/TestJobSubmission.java cccade4 
  test/org/apache/pig/test/TestPackage.java 6b197d1 
  test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a 
  test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728 
  test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f 
  test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d 
  test/org/apache/pig/test/data/GoldenFiles/MRC12.gld a5c5a1b 
  test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b 
  test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509 
  test/org/apache/pig/test/data/GoldenFiles/MRC15.gld cce2523 
  test/org/apache/pig/test/data/GoldenFiles/MRC16.gld c319c87 
  test/org/apache/pig/test/data/GoldenFiles/MRC17.gld 7e92e31 
  test/org/apache/pig/test/data/GoldenFiles/MRC18.gld d13e3ab 
  test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 569fd21 
  test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0 
  test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f 
  test/org/apache/pig/test/data/GoldenFiles/MRC4.gld e0bf184 
  test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 25dc2c3 
  test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 1b8e9c0 
  test/org/apache/pig/test/data/GoldenFiles/MRC8.gld f3beba8 
  test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c162b2d 

Diff: https://reviews.apache.org/r/15881/diff/


Testing
-------

ant test-commit passes, with the exception of TestExampleGenerator which is 
caused by the illustrate work not being finished. I also ran some manual 
scripts, which worked.


Thanks,

Mark Wagner

Review Request 15881: PIG-3591: Refactor POPackage

Reply via email to