-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15194/
-----------------------------------------------------------
(Updated Nov. 28, 2013, 12:41 a.m.)
Review request for pig, Cheolsoo Park, Daniel Dai, and Rohini Palaniswamy.
Changes
-------
Update with POPackage refactoring (see PIG-3595). There are still some items
from previous reviews that I have not completed.
Bugs: PIG-3527
https://issues.apache.org/jira/browse/PIG-3527
Repository: pig-git
Description
-------
Adds support for multiple LogicalInputs to the PigProcessor. This is done by
adding a new TezLoad interface which PhysicalOperators may implement. On the
backend, any operators implementing this interface will have the LogicalInput
attached to them. 2 implementations are included:
* POSimpleTezLoad which consumes a single MRInput
* POShuffleTezLoad which consumes one or more ShuffledMergedInputs.
The POShuffleTezLoad does a k-way merge of the shuffle inputs to package for
the operator pipeline. This required a change to the comparators used so that
the sort order remained consistent. There is also a fix to POForEach where it
was using the incorrect status code for signaling (although it produced the
same end result in the MR pipeline).
Diffs (updated)
-----
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/AccumulatorOptimizer.java
3638b5c
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
fc9261c
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java
9341153
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRUtil.java
93de6d5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
9580c26
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MultiQueryOptimizer.java
64f0ee1
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PhyPlanSetter.java
933363d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java
ddea99e
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java
5ea3fc7
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBooleanRawComparator.java
dfd4ebf
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBytesRawComparator.java
09397e5
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigCombiner.java
906930a
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDateTimeRawComparator.java
a87161f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigDoubleRawComparator.java
cbf457f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigFloatRawComparator.java
1d86e3f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java
eea5ce3
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigIntRawComparator.java
bb6c9df
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigLongRawComparator.java
b3ded76
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigSecondaryKeyComparator.java
5ad334b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTextRawComparator.java
022f37b
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java
866c39d
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleSortComparator.java
9724b9f
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizer.java
1578630
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/plans/POPackageAnnotator.java
47137d5
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/POSimpleTezLoad.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PhyPlanVisitor.java
abb16ff
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/PlanPrinter.java
ff82801
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/plans/XMLPhysicalPlanPrinter.java
892c26f
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/CombinerPackager.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/JoinPackager.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/LitePackager.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/MultiQueryPackager.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POCombinerPackage.java
9105a0e
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POJoinPackage.java
82f11ac
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMultiQueryPackage.java
d604174
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackage.java
86314d9
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPackageLite.java
c200715
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/Packager.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java
b860521
src/org/apache/pig/backend/hadoop/executionengine/tez/FileInputHandler.java
d29e330
src/org/apache/pig/backend/hadoop/executionengine/tez/InputHandler.java
d2298ca
src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java
68610dd
src/org/apache/pig/backend/hadoop/executionengine/tez/ShuffledInputHandler.java
d7b42b8
src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java
87b198f
src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
3dcefa9
src/org/apache/pig/backend/hadoop/executionengine/tez/TezLoad.java
PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/tez/TezPOPackageAnnotator.java
a57ac45
src/org/apache/pig/backend/hadoop/executionengine/util/CombinerOptimizerUtil.java
b163184
src/org/apache/pig/data/BinInterSedes.java b3ec51e
src/org/apache/pig/data/DefaultTuple.java 2e7ca5f
src/org/apache/pig/data/ReadOnceBag.java e2b3887
src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java
7112695
src/org/apache/pig/pen/IllustratorAttacher.java db9c707
src/org/apache/pig/pen/PhysicalPlanResetter.java f50fba7
test/e2e/pig/tests/tez.conf 4928a8e
test/org/apache/pig/test/TestJobSubmission.java cccade4
test/org/apache/pig/test/TestPackage.java 6b197d1
test/org/apache/pig/test/data/GoldenFiles/Cogroup.gld 35ed75a
test/org/apache/pig/test/data/GoldenFiles/MRC1.gld 0a34728
test/org/apache/pig/test/data/GoldenFiles/MRC10.gld b9fad0f
test/org/apache/pig/test/data/GoldenFiles/MRC11.gld 861608d
test/org/apache/pig/test/data/GoldenFiles/MRC12.gld a5c5a1b
test/org/apache/pig/test/data/GoldenFiles/MRC13.gld 0d4de2b
test/org/apache/pig/test/data/GoldenFiles/MRC14.gld eafa509
test/org/apache/pig/test/data/GoldenFiles/MRC15.gld cce2523
test/org/apache/pig/test/data/GoldenFiles/MRC16.gld c319c87
test/org/apache/pig/test/data/GoldenFiles/MRC17.gld 7e92e31
test/org/apache/pig/test/data/GoldenFiles/MRC18.gld d13e3ab
test/org/apache/pig/test/data/GoldenFiles/MRC19.gld 569fd21
test/org/apache/pig/test/data/GoldenFiles/MRC2.gld 97f0ed0
test/org/apache/pig/test/data/GoldenFiles/MRC3.gld 77a155f
test/org/apache/pig/test/data/GoldenFiles/MRC4.gld e0bf184
test/org/apache/pig/test/data/GoldenFiles/MRC6.gld 25dc2c3
test/org/apache/pig/test/data/GoldenFiles/MRC7.gld 1b8e9c0
test/org/apache/pig/test/data/GoldenFiles/MRC8.gld f3beba8
test/org/apache/pig/test/data/GoldenFiles/MRC9.gld c162b2d
Diff: https://reviews.apache.org/r/15194/diff/
Testing
-------
Manual testing and an e2e test has been added. Because of the comparator
change, some of the tests fail because of bag ordering.
Thanks,
Mark Wagner
