> On Sept. 3, 2013, 9:48 p.m., Mark Wagner wrote: > > src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java, line 29 > > <https://reviews.apache.org/r/13950/diff/1/?file=347549#file347549line29> > > > > Do we still need this when we have the DAG api from Tez? It seems > > strange to wrap Tez things in legacy MR APIs. If this is really needed, is > > it general enough to be included in the Tez project?
I kept TezJob (extension of Job) and JobControlCompiler because I thought Pig scripts would generate multiple MR* Tez DAGs, and we need to keep track of dependencies among them by JobControl structure. I guess you're thinking of building a giant DAG out of the entire Pig script. My question is, "Can we connect reduce vertices to mapper vertices using shuffle edges?" For eg, when I have MRR + MRR, can I submit it as a single DAG? Looking at Hive code, it looks like MRR + MRR will be submitted as two separate DAGs. Here is the comment in TezWork.java in Hive: * TezWork. This class encapsulates all the work objects that can be executed * in a single tez job. Currently it's basically a tree with MapWork at the * leaves and and ReduceWork in all other nodes. > On Sept. 3, 2013, 9:48 p.m., Mark Wagner wrote: > > src/org/apache/pig/backend/hadoop/executionengine/tez/MRROptimizer.java, > > line 26 > > <https://reviews.apache.org/r/13950/diff/1/?file=347543#file347543line26> > > > > I think this is related to the 'pigeon' flow of compilation. Is it > > still applicable for the new approach? You're probably right. TezCompiler can directly compile the physical plan into MRR. I will delete it. - Cheolsoo ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13950/#review25860 ----------------------------------------------------------- On Sept. 3, 2013, 9:14 p.m., Cheolsoo Park wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/13950/ > ----------------------------------------------------------- > > (Updated Sept. 3, 2013, 9:14 p.m.) > > > Review request for pig. > > > Bugs: PIG-3448 > https://issues.apache.org/jira/browse/PIG-3448 > > > Repository: pig-git > > > Description > ------- > > Adds skeleton classes that I think we need to implement for Tez backend. > > > Diffs > ----- > > build.xml 7e22192 > ivy.xml aa8f90a > ivy/libraries.properties 474edbd > src/META-INF/services/org.apache.pig.ExecType 7065767 > src/org/apache/pig/backend/hadoop/executionengine/tez/DagUtils.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/MRROptimizer.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/MapOper.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/ReduceOper.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecutionEngine.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezOpPlanVisitor.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperPlan.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezPrinter.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/13950/diff/ > > > Testing > ------- > > > Thanks, > > Cheolsoo Park > >
