[
https://issues.apache.org/jira/browse/YARN-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325539#comment-15325539
]
Xuan Gong commented on YARN-5237:
---------------------------------
Currently, we remove the finished container when we do the log aggregation
which works fine if rolling log aggregation is disabled. Because the log
aggregation only happens once.
But, it would cause the issue if rolling log aggregation is enabled. Because,
if we set includePattern/excludePattern, and we would not consider this during
the application is still running. If the container finishes, and we remove the
finished container, we would lose the chance to do the log aggregation (for
includePattern/excludePattern) when the app finishes.
{code}
this.pendingContainers.drainTo(pendingContainerInThisCycle);
Set<ContainerId> finishedContainers =
new HashSet<ContainerId>(pendingContainerInThisCycle);
// This container is finished, and all its logs have been uploaded,
// remove it from containerLogAggregators.
if (finishedContainers.contains(container)) {
containerLogAggregators.remove(container);
}
{code}
> Not all logs get aggregated with rolling log aggregation.
> ---------------------------------------------------------
>
> Key: YARN-5237
> URL: https://issues.apache.org/jira/browse/YARN-5237
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-5237.1.patch
>
>
> Steps to reproduce:
> 1) enable RM recovery
> 2) Run a sleep job
> 3) restart RM
> 4) kill the application
> We can not find that the logs for the first attempt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
