Ming Ma created YARN-4720:
-----------------------------
Summary: Skip unnecessary NN operations in log aggregation
Key: YARN-4720
URL: https://issues.apache.org/jira/browse/YARN-4720
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Ming Ma
Log aggregation service could have unnecessary NN operations in the following
scenarios:
* No new local log has been created since the last upload for the long running
service scenario.
* NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for
certain containers.
In the following code snippet, even though {{pendingContainerInThisCycle}} is
empty, it still creates the writer and then removes the file later. Thus it
introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't
aggregate logs for an app.
{noformat}
AppLogAggregatorImpl.java
......
writer =
new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp,
this.userUgi);
......
for (ContainerId container : pendingContainerInThisCycle) {
......
}
......
if (remoteFS.exists(remoteNodeTmpLogFileForApp)) {
if (rename) {
remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath);
} else {
remoteFS.delete(remoteNodeTmpLogFileForApp, false);
}
}
......
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
