Varun Saxena commented on YARN-2256:

[~zjshen], just to brief you on the issue; in our setup we were getting too 
many audit logs related to container events. We also found some other 
unnecessary logs(not required for debugging) appearing frequently. Had raised 
another JIRA for this. Anyways, so we internally we took up the task of 
cleaning up these logs. This also made a slight improvement in the throughput 
of running process(2.4.0)

To resolve the problem, one option was to remove these logs completely. But, we 
decided to support different log levels for audit logs so that if some customer 
requires these logs, we can enable them by merely changing log4j properties.
The scope of these 2 JIRAs' is indeed inter related. But I segregated them, 
because I wasnt sure if community would accept support for different log 
levels. We can decide if we need either one of these.

> Too many nodemanager and resourcemanager audit logs are generated
> -----------------------------------------------------------------
>                 Key: YARN-2256
>                 URL: https://issues.apache.org/jira/browse/YARN-2256
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-2256.patch
> Following audit logs are generated too many times(due to the possibility of a 
> large number of containers) :
> 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a 
> container
> 2. In RM - Audit logs corresponding to AM allocating a container and AM 
> releasing a container
> We can have different log levels even for NM and RM audit logs and move these 
> successful container related logs to DEBUG.

This message was sent by Atlassian JIRA

Reply via email to