Jason Lowe commented on YARN-3154:

Note that even LRS apps have issues if they don't do their own log rolling.  If 
I remember correctly, stdout and stderr files are setup by the container 
executor, and we'll have partial logs uploaded then deleted from the local 
filesystem, losing any subsequent logs to these files or any other files that 
aren't explicitly log rolled and filtered via a log aggregation context.

IMHO we need to make sure we do _not_ delete anything for a running app 
_unless_ it has a log aggregation context filter to tell us what is safe to 
upload and delete.  Without that information, we cannot tell if a log file is 
"live" and therefore going to be deleted too early.

> Should not upload partial logs for MR jobs or other "short-running' 
> applications 
> ---------------------------------------------------------------------------------
>                 Key: YARN-3154
>                 URL: https://issues.apache.org/jira/browse/YARN-3154
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>            Priority: Blocker
> Currently, if we are running a MR job, and we do not set the log interval 
> properly, we will have their partial logs uploaded and then removed from the 
> local filesystem which is not right.
> We only upload the partial logs for LRS applications.

This message was sent by Atlassian JIRA

Reply via email to