----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30299/#review70159 -----------------------------------------------------------
Just took a quick look at the first couple of files only. src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java <https://reviews.apache.org/r/30299/#comment115195> Stats must be collected for every Spark job. There are as many Spark jobs as there are POStore operators. So, we must collect stats for each POStore operator. src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java <https://reviews.apache.org/r/30299/#comment115196> please remove src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java <https://reviews.apache.org/r/30299/#comment115197> What is the purpose of connectSoftLink() ? Can we do this inside SparkCompiler.compile() itself ? src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkCompiler.java <https://reviews.apache.org/r/30299/#comment115198> please add javadoc - Mohit Sabharwal On Jan. 27, 2015, 1:27 a.m., kelly zhang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30299/ > ----------------------------------------------------------- > > (Updated Jan. 27, 2015, 1:27 a.m.) > > > Review request for pig, Mohit Sabharwal and Praveen R. > > > Repository: pig-git > > > Description > ------- > > @Mohit, I also added SparkPigStatsUtil.java( same with your SparkStatsUtil), > SparkPigStats.java and SparkJobStats.java in PIG-4393.patch. Maybe have some > conflictions. > > > Diffs > ----- > > > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkExecutionEngine.java > db152b5 > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > b15994d > > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkCompiler.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkCompilerException.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkOpPlanVisitor.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkOper.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkOperPlan.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/plan/SparkPOPackageAnnotator.java > PRE-CREATION > src/org/apache/pig/tools/pigstats/SparkPigStats.java PRE-CREATION > src/org/apache/pig/tools/pigstats/SparkStats.java fd45dd4 > src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkPigStatsUtil.java PRE-CREATION > > Diff: https://reviews.apache.org/r/30299/diff/ > > > Testing > ------- > > PIG-43741.patch is the initial patch. After testing in my jenkins, 66 new > failures are added. > > > Thanks, > > kelly zhang > >
