[ 
https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305159#comment-14305159
 ] 

Jason Lowe commented on YARN-3089:
----------------------------------

bq. the issue now happens for logs of all applications (and not just those that 
leverage YARN-2468)?

I think we're OK for normal apps.  It will try, and fail, to delete the 
individual log files, but then afterwards it issues a recursive delete on the 
log directory which does succeed.  We only found this problem because we saw 
delete tasks being leaked in the state store, and that's because errors from 
the container executor were escaping the try blocks.  See YARN-3090.

bq. For the changes in YARN-2468, we only delete the very old logs from HDFS. 
We do not delete any logs from NM.

The call of DeletionService.delete where basepaths are specified but subdir is 
empty/null is new with YARN-2468, specifically in this part of the code in 
AppLogAggregatorImpl.uploadLogsForContainers:

{code}
        this.delService.delete(this.userUgi.getShortUserName(), null,
          uploadedFilePathsInThisCycle
            .toArray(new Path[uploadedFilePathsInThisCycle.size()]));
{code}

The LCE wasn't properly handling the case when the basedirs arguments are _not_ 
directories.  (Based on the name of the argument, this isn't that surprising.)

The breakage we'll see is that on systems using the LCE the local delete 
triggered by the above code will not occur.  If the log directory is not going 
to be deleted shortly afterwards then we won't cleanup logs on the local disk 
in a timely manner after aggregation.

> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> ---------------------------------------------------------------------
>
>                 Key: YARN-3089
>                 URL: https://issues.apache.org/jira/browse/YARN-3089
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Eric Payne
>            Priority: Blocker
>         Attachments: YARN-3089.v1.txt, YARN-3089.v2.txt
>
>
> YARN-2468 added the deletion of individual logs that are aggregated, but this 
> fails to delete log files when the LCE is being used.  The LCE native 
> executable assumes the paths being passed are paths and the delete fails.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to