[
https://issues.apache.org/jira/browse/YARN-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15166575#comment-15166575
]
Ming Ma commented on YARN-4720:
-------------------------------
It seems that {{LogAggregationStatus.RUNNING}} implies the log aggregation
service is running, it doesn't necessarily mean NM actually aggregate any logs.
So if the long running service is running and hasn't generate any logs since it
starts, it is better to return {{LogAggregationStatus.RUNNING}}.
Yes, NM can send several {{LogAggregationReport}}s in the list which is
ordered; that is the API between NM and RM. Then on RM side, it will retrieve
all elements from the list.
> Skip unnecessary NN operations in log aggregation
> -------------------------------------------------
>
> Key: YARN-4720
> URL: https://issues.apache.org/jira/browse/YARN-4720
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Jun Gong
> Attachments: YARN-4720.01.patch, YARN-4720.02.patch
>
>
> Log aggregation service could have unnecessary NN operations in the following
> scenarios:
> * No new local log has been created since the last upload for the long
> running service scenario.
> * NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for
> certain containers.
> In the following code snippet, even though {{pendingContainerInThisCycle}} is
> empty, it still creates the writer and then removes the file later. Thus it
> introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't
> aggregate logs for an app.
>
> {noformat}
> AppLogAggregatorImpl.java
> ......
> writer =
> new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
> this.userUgi);
> ......
> for (ContainerId container : pendingContainerInThisCycle) {
> ......
> }
> ......
> if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
> if (rename) {
> remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
> } else {
> remoteFS.delete(remoteNodeTmpLogFileForApp, false);
> }
> }
> ......
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
