> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many
> > people launch hbase jobs from java action and instead of each user adding
> > killChildJobs doing it in JavaActionExecutor should be good. Currently the
> > configuration is not passed properly to distcp action. For killing child
> > jobs in distcp to work, we need to fix that or pass the configuration as
> > arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with
> > that tag. I think we can pass it as arg for now.
>
> Robert Kanter wrote:
> The reason I didn't add the killChildJobs to the Distcp action is that we
> don't have a "DistcpMain" to put it in. I didn't want to put the
> killChildJobs in the calling class LauncherMapper because (a) we handle this
> differently for the MR action and (b) someone writing a custom action (or us
> in a future action) may also want to handle this differently. I guess I
> could write a wrapper "DistcpMain" to call killChildJobs...
>
> For the Java and Shell actions, they don't necessarily launch any child
> jobs. And in general, we don't implement some of the "Oozie magic" stuff for
> the Java and Shell actions, leaving it up to the user instead (for example,
> propagating the action's <configuration> or the delegation token). The Java
> action also has the same problem as the Distcp action, in that there's no
> "JavaMain". What if I added documentation on how to call killChildJobs so
> the user can do this in their Main?
>
> Rohini Palaniswamy wrote:
> Many users launch child jobs in their java actions (especially with
> hbase). Would prefer as much less work for user as possible. If you are
> writing a DistcpMain, you can just write a JavaMain that calls killChildJobs
> and make DistcpMain just extend that. That way it will just work without
> users having to bother about it and writing different code for different
> versions of hadoop.
>
> Robert Kanter wrote:
> I can add a wrapper for Java and Distcp that does the killing. Though
> for the Java action, it will still be up to the user to propagate the
> actionConf prepared by Oozie, or at least just set the tag property.
>
> Rohini Palaniswamy wrote:
> Sounds good and thanks. Most of them usually propagate the actionConf and
> would benefit from this.
>
> Robert Kanter wrote:
> Getting this to work with DistCp is trickier than I thought. I tried
> passing the tags property as a -D argument, but that didn't work (in fact, it
> caused an error). Looking at DistCp's code, we could try to invoke it
> programmatically from a "DistCpMain" instead of JavaMain; this would allow us
> to pass it a Configuration. However, DistCp v1 (Hadoop 1) and v2 (Hadoop
> 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an
> additional argument), so the "DistCpMain" would have to use Reflection and
> check which constructor is available. MAPREDUCE-5081 did backport v2 to
> Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>
> So we have a few options:
> (1) Don't support the tags with DistCp
> (2) Use a "DistCpMain"
> (2a) Use reflection to call the appropriate constructor and run it
> programmatically
> (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of
> Hadoop
>
> I think we should do (1) for now so we can get the tags stuff in and
> create a follow-up JIRA to do (2b), which will require a lot of additional
> testing.
>
> Thoughts?
Option 3:
<arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information is
currently passed to Distcp from the distcp action.
LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments
handle the main method arguments. We need to introduce it somewhere there for
Distcp. But it is hacky and don't like it much. Just pointing out as an option.
Your option 2 is cleaner and like it better. We definitely need to fix
DistcpMain to call the constructor instead of the main() method to pass the
JobConf. Users have been passing even the queue information as above because
the distcp job does not get the action configuration. We had this reported as
an issue, but have always gone with the <arg> workaround. So option 2 will fix
more issues. We can definitely do (2a). (2b) of course can be a different jira.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------
On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
>
> (Updated March 4, 2014, 3:11 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1722
> https://issues.apache.org/jira/browse/OOZIE-1722
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1722
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java
> fcf69e7
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> 68d77a8
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
> 07de214
>
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> 7841076
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7
> core/src/test/java/org/apache/oozie/service/TestShareLibService.java
> e7d7a26
>
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
> 41e6205
> hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION
>
> hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
> PRE-CREATION
> hadooplibs/pom.xml 35875c8
> pom.xml 1223ec2
> release-log.txt f6810bc
>
> sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java
> 25b1a23
> sharelib/hive/pom.xml e355d97
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java
> 8fa5ec8
> sharelib/oozie/pom.xml c70582d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java
> PRE-CREATION
>
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
> b5fa4a1
>
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java
> 6230e64
> sharelib/pig/pom.xml a2bbeaf
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java
> 9c1e659
> sharelib/sqoop/pom.xml a962565
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java
> e45c8b2
>
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/18653/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
