----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23787/#review48753 -----------------------------------------------------------
trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/optimizers/LoaderProcessor.java <https://reviews.apache.org/r/23787/#comment85501> Actually change is simple and should not be necessary to open another jira. Can you get Credentials from the conf after casting it to (JobConf) and set it on tezOp here and then add that to dag.getCredentials() in TezDagBuilder. - Rohini Palaniswamy On July 25, 2014, 1:36 a.m., Daniel Dai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/23787/ > ----------------------------------------------------------- > > (Updated July 25, 2014, 1:36 a.m.) > > > Review request for pig. > > > Bugs: PIG-4057 > https://issues.apache.org/jira/browse/PIG-4057 > > > Repository: pig > > > Description > ------- > > Summary of changes: > 1. Take tez parallelism estimation out from TezDagBuilder to > ParallelismSetter, so we can get estimated parallelism of the cross before we > creating vertex of GFCross > 2. Take InputSplit generate out from TezDagBuilder to LoaderProcessor, since > we need to know the parallelism of maps before ParallelismSetter > 3. set pig.cross.parallelism.hint.(operator_key) in conf > * In tez, this is done when we encounter cross vertex > * In MR, this is done when we encounter the first GFCross > 4. GFCross will use pig.cross.parallelism.hint.(operator_key) to determine > the #partition > > > Diffs > ----- > > trunk/src/org/apache/pig/PigConfiguration.java 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POGlobalRearrange.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java > 1613328 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/optimizers/LoaderProcessor.java > PRE-CREATION > > trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/optimizers/ParallelismSetter.java > PRE-CREATION > trunk/src/org/apache/pig/impl/builtin/GFCross.java 1613328 > > trunk/src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java > 1613328 > trunk/test/e2e/pig/tests/nightly.conf 1613328 > trunk/test/org/apache/pig/test/TestGFCross.java 1613328 > > Diff: https://reviews.apache.org/r/23787/diff/ > > > Testing > ------- > > > Thanks, > > Daniel Dai > >
