[
https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404213#comment-13404213
]
Jie Li commented on PIG-2780:
-----------------------------
The jobControlException is different from the job exception. Pig submit jobs in
batches and check exceptions at the end of each batch. Jobs without
inter-dependency can run in parallel in the same batch, otherwise they are in
different batches. Usually the number of jobs in one batch is very small, so
checking exception at the end looks acceptable.
If you specify "-stop_on_failure" in the command line option, then Pig will
stop after the failed batch.
If it doesn't satisfy your need, i.e. you also want Pig to stop on failure
within batch, then please keep this jira open.
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
>
> Right now MapReduceLauncher caches the job exception in jobControlException
> and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing
> exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until
> other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
