> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, > > line 1483 > > <https://reviews.apache.org/r/27614/diff/2/?file=757470#file757470line1483> > > > > type should be the second argument. i.e. > > > > LOG.debug("class for [{0}] Action: [{1}]", type, function);
Sure will fix it > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, > > line 32 > > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line32> > > > > Can we name this something else? It's easy to confuse it with > > LauncherMapper.CONF_OOZIE_ACTION_MAIN_CLASS This is not required any more, i will call SparkSubmit directly instead of using reflection > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, > > line 34 > > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line34> > > > > Do we actually need to use this? In my experience, it causes nothing > > but trouble... Yes it is needed due to some conflicts in dependencies > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, > > lines 35-41 > > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line35> > > > > These should have more unique names (like we do in the other actions). > > This helps prevent collisions as they go into the actionConf. > > > > e.g. > > oozie.spark.spark > > oozie.spark.master > > oozie.spark.mode > > etc Sure will do > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, > > line 78 > > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line78> > > > > We have this for DistCp because the classes are different between > > different versions of MapReduce and there's also two versions of DistCp. > > I'm not familiar enough with Spark, but is there any reason for a user to > > use a different Spark class? > > > > If not, then you can get rid of this to simplify things. In fact, > > SparkMain should subclass LauncherMain instead of JavaMain, and you can > > call SparkSubmit.main(...) directly instead of using reflection. Look at > > HiveMain or SqoopMain for examples. I thought later somenbody can change the Spark job launcher class , so that we can take it from properties like distcp. Anyway i will remove it > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, > > line 85 > > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line85> > > > > Do we actually need to use this property? In my experience, it causes > > nothing but trouble... Yes it is needed > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > sharelib/spark/pom.xml, line 41 > > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line41> > > > > whitespace will fix > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > sharelib/spark/pom.xml, line 45 > > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line45> > > > > version numbers should go in the root pom All other actions are using guava 11.0.2 , to run spark action we need 14.0.1 for the reason i dont want to change in root pom > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > sharelib/spark/pom.xml, line 51 > > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line51> > > > > versions numbers should go in the root pom will do > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, > > line 29 > > <https://reviews.apache.org/r/27614/diff/2/?file=757476#file757476line29> > > > > See earlier comment where I explain that this should subclass > > LauncherMain and other changes Sure will change > On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote: > > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, > > line 100 > > <https://reviews.apache.org/r/27614/diff/2/?file=757476#file757476line100> > > > > Most of the other actions have some additional logic to send > > information back to Oozie (e.g. launched child job ids, other stats). I'm > > not sure but I believe Spark jobs have an ID, right? Is it possible to > > return this back to Oozie as the child job? Look at some of the other > > actions to see how we do this. Yes spark jobs have application id when they are running in spark cluster , according to my knowledge i don't think there is way to get the application id from the cluster . - pavan kumar ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27614/#review60961 ----------------------------------------------------------- On Nov. 11, 2014, 6:35 a.m., pavan kumar kolamuri wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27614/ > ----------------------------------------------------------- > > (Updated Nov. 11, 2014, 6:35 a.m.) > > > Review request for oozie and shwethags. > > > Bugs: OOZIE-1983 > https://issues.apache.org/jira/browse/OOZIE-1983 > > > Repository: oozie-git > > > Description > ------- > > Add spark action executor in oozie. Spark jobs can be run using oozie > > > Diffs > ----- > > client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b > client/src/main/resources/spark-action-0.1.xsd PRE-CREATION > core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java > 42f2965 > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > 7349d3f > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java > PRE-CREATION > core/src/main/resources/oozie-default.xml 17155a1 > pom.xml 1e79186 > sharelib/pom.xml aa479a8 > sharelib/spark/pom.xml PRE-CREATION > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java > PRE-CREATION > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java > PRE-CREATION > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java > PRE-CREATION > src/main/assemblies/sharelib.xml 4a46b90 > webapp/pom.xml 35776c5 > > Diff: https://reviews.apache.org/r/27614/diff/ > > > Testing > ------- > > Both unit testing and end to end testing done > > > Thanks, > > pavan kumar kolamuri > >
