> On July 25, 2014, 6:28 p.m., Rohini Palaniswamy wrote: > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java, > > line 425 > > <https://reviews.apache.org/r/23787/diff/4/?file=641636#file641636line425> > > > > pig.inpTargets not required?
It is only used by PigInputFormat.getSplits. Since we move PigInputFormat.getSplits to LoaderProcessor, we don't need "pig.inpTargets" in TezDagBuilder. > On July 25, 2014, 6:28 p.m., Rohini Palaniswamy wrote: > > trunk/src/org/apache/pig/PigConfiguration.java, line 274 > > <https://reviews.apache.org/r/23787/diff/4/?file=641632#file641632line274> > > > > It is a internal setting and not user facing one. We should probably > > create a new class called PigInternalConfiguration for those. > > > > Can also remove "hint" from the name as it is used as is. > > Cheolsoo Park wrote: > We have PigConstants fot internal configurations. We can use that, no? Seems we need to clean it up and diverge to PigInternalConfiguration if needed. Probably not proper to just piggyback in this patch. - Daniel ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23787/#review48747 ----------------------------------------------------------- On July 25, 2014, 11:33 p.m., Daniel Dai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23787/ > ----------------------------------------------------------- > > (Updated July 25, 2014, 11:33 p.m.) > > > Review request for pig. > > > Bugs: PIG-4057 > https://issues.apache.org/jira/browse/PIG-4057 > > > Repository: pig > > > Description > ------- > > Summary of changes: > 1. Take tez parallelism estimation out from TezDagBuilder to > ParallelismSetter, so we can get estimated parallelism of the cross before we > creating vertex of GFCross > 2. Take InputSplit generate out from TezDagBuilder to LoaderProcessor, since > we need to know the parallelism of maps before ParallelismSetter > 3. set pig.cross.parallelism.hint.(operator_key) in conf > * In tez, this is done when we encounter cross vertex > * In MR, this is done when we encounter the first GFCross > 4. GFCross will use pig.cross.parallelism.hint.(operator_key) to determine > the #partition > > > Diffs > ----- > > trunk/src/org/apache/pig/PigConfiguration.java 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POGlobalRearrange.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java > 1613525 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/optimizers/LoaderProcessor.java > PRE-CREATION > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/optimizers/ParallelismSetter.java > PRE-CREATION > trunk/src/org/apache/pig/impl/builtin/GFCross.java 1613525 > > trunk/src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java > 1613525 > trunk/test/e2e/pig/tests/nightly.conf 1613525 > trunk/test/org/apache/pig/test/TestGFCross.java 1613525 > > Diff: https://reviews.apache.org/r/23787/diff/ > > > Testing > ------- > > > Thanks, > > Daniel Dai > >
