[ 
https://issues.apache.org/jira/browse/YARN-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355887#comment-14355887
 ] 

Xuan Gong commented on YARN-3154:
---------------------------------

bq. do the long running applications such as HBase on YARN using Slider need to 
do anything to make sure that partial logs are uploaded?

[~sumitmohanty] Sorry for the late reply. Yes, we need to change some 
configurations/setting for ApplicationSubmissionContext.

Here is a scenario which can explain the purpose of this ticket:
In MapReduce, we will create stdout, stderr, and syslog for every containers. 
And since the MapReduce job is relatively short (compared with the long running 
applications), it does not make sense to upload those logs partially unless the 
users really want to.
So, the old include_pattern/exclude_pattern in ASC will be used to indicate 
which log files need to be aggregated explicitly at app finish.
and we introduce two additional parameter is ASC which is more related to long 
running applications, such as HBase on YARN.
{code}
rolled_logs_include_pattern 
rolled_logs_exclude_pattern
{code}
If we want the logs be uploaded (partial logs) while the app is running, we 
should use these two newly instroduced parameters.

For the  HBase on YARN using Slider case, after the patch, we need to switch 
the values from old include_pattern/exclude_pattern to new 
rolled_logs_include_pattern/rolled_logs_exclude_pattern

> Should not upload partial logs for MR jobs or other "short-running' 
> applications 
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-3154
>                 URL: https://issues.apache.org/jira/browse/YARN-3154
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>            Priority: Blocker
>         Attachments: YARN-3154.1.patch, YARN-3154.2.patch, YARN-3154.3.patch
>
>
> Currently, if we are running a MR job, and we do not set the log interval 
> properly, we will have their partial logs uploaded and then removed from the 
> local filesystem which is not right.
> We only upload the partial logs for LRS applications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to