----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61265/ -----------------------------------------------------------
(Updated Aug. 1, 2017, 10:41 p.m.)
Review request for pig, Daniel Dai and Koji Noguchi.
Changes
-------
Did some cleanup and fixed some e2e failures. There a couple more to fix
Bugs: PIG-5256
https://issues.apache.org/jira/browse/PIG-5256
Repository: pig
Description (updated)
-------
For each POForEach or POFilter operator in the plan, a corresponding class is
created by inlining code for the input plans.
Bytecode is generated directly through asm APIs and the generated class files
are shipped to the tasks as a jar. On the frontend, the generated classes are
dynamically added to the PigContext classloader. The explain command now
decompiles the class files in the explain output directory. Refer to the golden
files in the test for examples of generated code.
TODO items to be addressed in a separate jira:
1) PIG-5279 - Support for MR and Spark. Currently only done for Tez. Will also
add documentation in this jira
2) Support for Accumulator
3) Support for CROSS
4) Run the optimizer on combiner plans
Diffs (updated)
-----
http://svn.apache.org/repos/asf/pig/trunk/build.xml 1803384
http://svn.apache.org/repos/asf/pig/trunk/ivy.xml 1803384
http://svn.apache.org/repos/asf/pig/trunk/ivy/libraries.properties 1803384
http://svn.apache.org/repos/asf/pig/trunk/shade/pom.xml PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigConfiguration.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigServer.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/AsmUtil.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/CodeGenerator.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/FilterCodeGenerator.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/ForEachCodeGenerator.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Add.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Divide.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/EqualToExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GTOrEqualToExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GreaterThanExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LTOrEqualToExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LessThanExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Mod.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Multiply.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/NotEqualToExpr.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POCast.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POProject.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/PORegexp.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/PORelationToExprProject.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Subtract.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/PODistinct.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFilter.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLimit.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POOptimizedForEach.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSortedDistinct.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezPlanContainer.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/RuntimeCodeGenOptimizer.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/InvokerGenerator.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/data/DataType.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/PigContext.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/io/FileLocalizer.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/plan/OperatorPlan.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/ClassDecompiler.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/JarManager.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/logical/expression/ExpToPhyTranslationVisitor.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/logical/expression/UserFuncExpression.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/parser/LogicalPlanBuilder.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/tools/grunt/GruntParser.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestByteCodeGen.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestProjectStarRangeInUdf.java
1803384
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/POFilter_scope_12.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/POForEach_scope_11.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/plan.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POFilter_scope_14.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POFilter_scope_9.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POForEach_scope_7.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/plan.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_20.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_32.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_9.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/plan.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachUDFEval/POForEach_scope_35.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachUDFEval/plan.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POFilter_scope_20.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POFilter_scope_48.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_14.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_26.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_37.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_42.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_54.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_61.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_67.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_7.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_70.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/POForEach_scope_76.java.gld
PRE-CREATION
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testNestedForeach/plan.gld
PRE-CREATION
Diff: https://reviews.apache.org/r/61265/diff/2/
Changes: https://reviews.apache.org/r/61265/diff/1-2/
Testing (updated)
-------
Running full suite of unit and e2e tests with pig.opt.bytecode=true. Will
update soon
Thanks,
Rohini Palaniswamy
