Jason Lowe commented on YARN-3090:

Thanks for the patch, Varun!  I kicked the tires on this patch and discovered 
it doesn't work.  I forced the container executor to throw a runtime exception 
but it wasn't logged.

It turns out I was mistaken in my earlier analysis.  The afterExecute method 
normally does not receive any exception because it only reports exceptions the 
_FutureTask run method_ throws, and normally that doesn't throw anything.  It 
stores any exception thrown by the underlying Runnable and almost always throws 
nothing itself.  Therefore afterExecute sees no exception even if one escaped 
during deletion, and we log nothing.  I mistakenly assumed that the thread pool 
executor would pass any exception stored by the FutureTask to afterExecute, but 
that's not the case.

To log the escaped exceptions we'd need to do something like this in 

    protected void afterExecute(Runnable task, Throwable exception) {
      if (task instanceof FutureTask<?>) {
        FutureTask<?> futureTask = (FutureTask<?>) task;
        if (!futureTask.isCancelled()) {
          try {
          } catch (ExecutionException e) {
            exception = e.getCause();
      if (exception != null) {
        LOG.warn("Exception during execution of task in DeletionService",

A couple of other nits on the patch:
* The new class should be private
* The log message should be an ERROR rather than a WARN since this should only 
occur when an exception escaped the exception handling already in place for 
deletion.  Those kinds of exceptions are usually pretty bad (like NPEs).

> DeletionService can silently ignore deletion task failures
> ----------------------------------------------------------
>                 Key: YARN-3090
>                 URL: https://issues.apache.org/jira/browse/YARN-3090
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.1.1-beta
>            Reporter: Jason Lowe
>            Assignee: Varun Saxena
>         Attachments: YARN-3090.001.patch, YARN-3090.002.patch
> If a non-I/O exception occurs while the DeletionService is executing a 
> deletion task then it will be silently ignored.  The exception bubbles up to 
> the thread workers of the ScheduledThreadPoolExecutor which simply attaches 
> the throwable to the Future that was returned when the task was scheduled.  
> However the thread pool is used as a fire-and-forget pool, so nothing ever 
> looks at the Future and therefore the exception is never logged.

This message was sent by Atlassian JIRA

Reply via email to