[
https://issues.apache.org/jira/browse/HADOOP-15790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413430#comment-17413430
]
Ayush Saxena commented on HADOOP-15790:
---------------------------------------
I think the problem still holds, But the patch doesn't apply now. Couple of
things have changed since last.
One thing to note is we have to check whether the job is blocking, if it is
non-blocking distcp, then we can't do this success check. Guess an additional
\{{context.shouldBlock()}} in the existing if check should make things work
> distcp app should fail if m/r job fails
> ---------------------------------------
>
> Key: HADOOP-15790
> URL: https://issues.apache.org/jira/browse/HADOOP-15790
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
> Affects Versions: 3.0.0-alpha1, 3.2.0
> Reporter: David Rosenstrauch
> Priority: Major
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5549-001.patch, MAPREDUCE-5549-002.patch
>
>
> I run distcpv2 in a scripted manner. The script checks if the distcp step
> fails and, if so, aborts the rest of the script. However, I ran into an
> issue today where the distcp job failed, but my calling script went on its
> merry way.
> Digging into the code a bit more (at
> https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java),
> I think I see the issue: the distcp app is not returning an error exit code
> to the shell when the distcp job fails. This is a big problem, IMO, as it
> prevents distcp from being successfully used in a scripted environment. IMO,
> the code should change like so:
> Before:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
> public int run(String[] argv) {
> //...
> try {
> execute();
> } catch (InvalidInputException e) {
> LOG.error("Invalid input: ", e);
> return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
> LOG.error("Duplicate files in input path: ", e);
> return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
> LOG.error("Exception encountered ", e);
> return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
> }
> //...
> {code}
> After:
> {code:title=org.apache.hadoop.tools.DistCp.java}
> //...
> public int run(String[] argv) {
> //...
> Job job = null;
> try {
> job = execute();
> } catch (InvalidInputException e) {
> LOG.error("Invalid input: ", e);
> return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
> LOG.error("Duplicate files in input path: ", e);
> return DistCpConstants.DUPLICATE_INPUT;
> } catch (Exception e) {
> LOG.error("Exception encountered ", e);
> return DistCpConstants.UNKNOWN_ERROR;
> }
> if (job.isSuccessful()) {
> return DistCpConstants.SUCCESS;
> }
> else {
> return DistCpConstants.UNKNOWN_ERROR;
> }
> }
> //...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
