[
https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14306220#comment-14306220
]
Xuan Gong commented on YARN-3089:
---------------------------------
bq. The call of DeletionService.delete where basepaths are specified but subdir
is empty/null is new with YARN-2468, specifically in this part of the code in
AppLogAggregatorImpl.uploadLogsForContainers:
Thanks for explanation. Jason.
bq. do you still want this committed to 2.6.1 given it should only affect LRS
apps?
For normal applications, the application logs directory will be deleted after
the application is finished. As Jason mentioned, this action gets succeed.
For the LRS applications, We deleted the logs which is already uploaded.
{code}
this.delService.delete(this.userUgi.getShortUserName(), null,
uploadedFilePathsInThisCycle
.toArray(new Path[uploadedFilePathsInThisCycle.size()]));
{code}
So, whether it can affect the LRS apps, my answer is "depends on". It is users'
responsibility to roll over their logs. YARN will upload and clean the logs
which exist in the log directory.
Currently, it depends on how users want to roll-over their application logs and
set up the LogAggregationContext.
For example, If the users roll-over the logs with some pattern, say "log.1,
log.2, ..., etc", and set up the includePattern in LogAggregationContext with
something like "log.(\\d+)". So, YARN will only upload the logs (log.1, log.2),
and delete the logs (log.1, log.2) afterward. In this case, *i do not think
this LRS app is affected at all*.
But if the user does not set any includePattern/excludePattern in
LogAggregationContext, and does not want to roll-over the logs, all container
logs will be written into the files with the same name(run the command, such as
1>>stdout, 2>>stderr ). After we aggregate the logs into HDFS, the
stdout/stderr will be deleted. *In this case, this LRS app is affected.*
> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> ---------------------------------------------------------------------
>
> Key: YARN-3089
> URL: https://issues.apache.org/jira/browse/YARN-3089
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Jason Lowe
> Assignee: Eric Payne
> Priority: Blocker
> Attachments: YARN-3089.v1.txt, YARN-3089.v2.txt, YARN-3089.v3.txt
>
>
> YARN-2468 added the deletion of individual logs that are aggregated, but this
> fails to delete log files when the LCE is being used. The LCE native
> executable assumes the paths being passed are paths and the delete fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
