----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14601/#review26956 -----------------------------------------------------------
I played around with this this afternoon and noticed a few things. I haven't gone through all of it yet. src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java <https://reviews.apache.org/r/14601/#comment52506> We'll also need to set some environment variables (I think CLASSPATH at a minimal). Check out MRHelpers src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java <https://reviews.apache.org/r/14601/#comment52504> This returns a configuration w/o defaults, but the TezJobControlCompiler turns it into a TezConfiguration, so it needs the tez-site items as well. I added a toConfiguration(Props, bool) that optionally includes defaults. I'm also getting error messages when I close the session about killing the ExecutionEngine. I haven't looked into that, though. - Mark Wagner On Oct. 12, 2013, 1:19 a.m., Cheolsoo Park wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14601/ > ----------------------------------------------------------- > > (Updated Oct. 12, 2013, 1:19 a.m.) > > > Review request for pig, Daniel Dai, Mark Wagner, and Rohini Palaniswamy. > > > Bugs: PIG-3502 > https://issues.apache.org/jira/browse/PIG-3502 > > > Repository: pig-git > > > Description > ------- > > Initial implementation of TezLauncher. The patch includes the following > changes: > - Refactors Launcher and MapReduceLauncher. Basically, I moved what's > MR-specific to MapReduceLauncher and kept what's common in Launcher. > - Removes kill() method from ExecutionEngine interface. It was used merely by > Launcher, and it didn't seem useful. > - Implements the DAG submission of TezLauncher. Basically, I overrode the > submit() method in TezJob (extension of ControlledJob) that calls > DAGClient.submit(). > - Adds a skeleton TezStats class that extends PigStats. This class needs to > be implemented to monitor job status after submission. > - Adds a unit test called TestTezLauncher. For now, all it does is to compile > queries and submit a DAG. More meaningful unit tests will be added after > TezStats is implemented. > - Adds a TezLocalExecType. To start up a YARN mini cluster, I needed > non-local Tez mode. So I created "tez_local" and "tez" modes. > - Whitespace and indentation fixes. > > > Diffs > ----- > > build.xml 0439082 > src/META-INF/services/org.apache.pig.ExecType e995cd5 > src/org/apache/pig/backend/executionengine/ExecutionEngine.java 7d480d7 > src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java > b886f9d > src/org/apache/pig/backend/hadoop/executionengine/Launcher.java 77543c2 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java > f3be738 > src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java > 5414906 > src/org/apache/pig/backend/hadoop/executionengine/tez/TezExecType.java > 8315e76 > src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java 04ffba0 > > src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobControlCompiler.java > b7d4d98 > src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java > ff9e950 > src/org/apache/pig/backend/hadoop/executionengine/tez/TezLocalExecType.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/tez/TezStats.java > PRE-CREATION > src/org/apache/pig/impl/PigContext.java 225526f > src/org/apache/pig/impl/util/Utils.java 3720c94 > test/org/apache/pig/tez/TestTezCompiler.java b9e6b48 > test/org/apache/pig/tez/TestTezJobControlCompiler.java f14d49d > test/org/apache/pig/tez/TestTezLauncher.java PRE-CREATION > > Diff: https://reviews.apache.org/r/14601/diff/ > > > Testing > ------- > > TestTezLauncher is added. > > > Thanks, > > Cheolsoo Park > >
