> On Jan. 30, 2014, 7:53 p.m., Daniel Dai wrote: > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java, > > line 1462 > > <https://reviews.apache.org/r/17529/diff/1/?file=454253#file454253line1462> > > > > What's the reason for this change? lr should be already constructed, > > right?
lr was just a plain POLOcalRearrangeTez without plan (only Constant(DummyVal) in its plan). Since we removed the LR in the Partition vertex and used POIdentityInOut which does not process key values, that had to be done moved to the input vertex. See TEZC16.gld for how orderby plan has changed. - Rohini ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17529/#review33243 ----------------------------------------------------------- On Jan. 30, 2014, 8:28 a.m., Rohini Palaniswamy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/17529/ > ----------------------------------------------------------- > > (Updated Jan. 30, 2014, 8:28 a.m.) > > > Review request for pig, Cheolsoo Park and Daniel Dai. > > > Bugs: PIG-3732 > https://issues.apache.org/jira/browse/PIG-3732 > > > Repository: pig > > > Description > ------- > > Orderby has 4 vertices and changes done are as below. > > Load Vertex -> Partitioner Vertex > - Was RoundRobinPartitioner with sorted shuffle and parallelism of > Partitioner Vertex was same as reducer vertex (i.e PARALLEL clause). Now > ONE_TO_ONE unsorted edge between Load Vertex and Partitioner Vertex with > Partitioner Vertex having same parallelism as Load Vertex. Will get the > performance numbers for both cases by Friday. > Load Vertex -> Sampler Vertex > Sampler Vertex -> Partitioner Vertex (Broadcast edge) > - The POPackage->POForeach->POLocalRearrange in Partitioner Vertex has > been replaced by POIdentityInOutTez > Partitioner Vertex -> Reducer Vertex > > Need to attempt this for Skewed Join as well. > > > This patch also sets credential on DAG which is required after TEZ-395 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java > PRE-CREATION > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POLocalRearrangeTez.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDAG.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/NullablePartitionWritable.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/PigNullableWritable.java > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC16.gld > 1562426 > > http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld > 1562426 > > Diff: https://reviews.apache.org/r/17529/diff/ > > > Testing > ------- > > test-tez and tez.conf e2e tests pass > > > Thanks, > > Rohini Palaniswamy > >
