> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote: > > We need to add the killChildJobs in Java, Shell and Distcp as well. Many > > people launch hbase jobs from java action and instead of each user adding > > killChildJobs doing it in JavaActionExecutor should be good. Currently the > > configuration is not passed properly to distcp action. For killing child > > jobs in distcp to work, we need to fix that or pass the configuration as > > arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with > > that tag. I think we can pass it as arg for now. > > Robert Kanter wrote: > The reason I didn't add the killChildJobs to the Distcp action is that we > don't have a "DistcpMain" to put it in. I didn't want to put the > killChildJobs in the calling class LauncherMapper because (a) we handle this > differently for the MR action and (b) someone writing a custom action (or us > in a future action) may also want to handle this differently. I guess I > could write a wrapper "DistcpMain" to call killChildJobs... > > For the Java and Shell actions, they don't necessarily launch any child > jobs. And in general, we don't implement some of the "Oozie magic" stuff for > the Java and Shell actions, leaving it up to the user instead (for example, > propagating the action's <configuration> or the delegation token). The Java > action also has the same problem as the Distcp action, in that there's no > "JavaMain". What if I added documentation on how to call killChildJobs so > the user can do this in their Main? > > Rohini Palaniswamy wrote: > Many users launch child jobs in their java actions (especially with > hbase). Would prefer as much less work for user as possible. If you are > writing a DistcpMain, you can just write a JavaMain that calls killChildJobs > and make DistcpMain just extend that. That way it will just work without > users having to bother about it and writing different code for different > versions of hadoop. > > Robert Kanter wrote: > I can add a wrapper for Java and Distcp that does the killing. Though > for the Java action, it will still be up to the user to propagate the > actionConf prepared by Oozie, or at least just set the tag property. > > Rohini Palaniswamy wrote: > Sounds good and thanks. Most of them usually propagate the actionConf and > would benefit from this.
Getting this to work with DistCp is trickier than I thought. I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error). Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration. However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available. MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1 So we have a few options: (1) Don't support the tags with DistCp (2) Use a "DistCpMain" (2a) Use reflection to call the appropriate constructor and run it programmatically (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing. Thoughts? - Robert ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18653/#review35947 ----------------------------------------------------------- On March 4, 2014, 3:11 a.m., Robert Kanter wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18653/ > ----------------------------------------------------------- > > (Updated March 4, 2014, 3:11 a.m.) > > > Review request for oozie. > > > Bugs: OOZIE-1722 > https://issues.apache.org/jira/browse/OOZIE-1722 > > > Repository: oozie-git > > > Description > ------- > > See https://issues.apache.org/jira/browse/OOZIE-1722 > > > Diffs > ----- > > core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java > fcf69e7 > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > 68d77a8 > core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java > 07de214 > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > 7841076 > core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 > core/src/test/java/org/apache/oozie/service/TestShareLibService.java > e7d7a26 > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > 41e6205 > hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION > > hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > PRE-CREATION > hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION > > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > PRE-CREATION > hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION > > hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > PRE-CREATION > hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION > > hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > PRE-CREATION > hadooplibs/pom.xml 35875c8 > pom.xml 1223ec2 > release-log.txt f6810bc > > sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java > 25b1a23 > sharelib/hive/pom.xml e355d97 > sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java > 8fa5ec8 > sharelib/oozie/pom.xml c70582d > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java > b5fa4a1 > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java > 6230e64 > sharelib/pig/pom.xml a2bbeaf > sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java > 9c1e659 > sharelib/sqoop/pom.xml a962565 > sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java > e45c8b2 > > sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/18653/diff/ > > > Testing > ------- > > > Thanks, > > Robert Kanter > >
