Varun Saxena commented on YARN-2256:

bq. Someone correct me if I'm wrong, but I'm fairly certain that the intent of 
this information is to be the equivalent of the HDFS audit log. In other words, 
setting these to debug completely defeats the purpose. Instead, I suspect the 
real culprit is that the log4j settings are wrong for the node manager process.

[~aw], the issue raised was basically for both NM and RM. I have updated the 
description to reflect that. The issue here is that some of the container 
related operations' audit logs in both NM and RM are too frequent and too many. 
This may impact performance as well.

Now, there are 2 solutions possible, either remove these logs or change the log 
level, so that they do not occur in live environment and can be opened only 
when required. 
As I wasnt sure if these audit logs have to be removed or not, I changed the 
log level for some of these logs in RM and all of them in NM. To ensure this, I 
supported printing of audit logs at different levels, as is done in HBase (as 
per my info). This is handled as part of YARN-2287

Now for NM, you are correct, Log level can be changed in log4j properties to 
suppress these logs if required. But for RM, as not all logs have to be 
suppressed, this cant be done. So to be consistent, I added log levels for both 
NM and RM. 

If its agreeable to remove these audit logs, that can be a possible solution as 
well. Pls suggest. 

> Too many nodemanager and resourcemanager audit logs are generated
> -----------------------------------------------------------------
>                 Key: YARN-2256
>                 URL: https://issues.apache.org/jira/browse/YARN-2256
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, resourcemanager
>    Affects Versions: 2.4.0
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>         Attachments: YARN-2256.patch
> Following audit logs are generated too many times(due to the possibility of a 
> large number of containers) :
> 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a 
> container
> 2. In RM - Audit logs corresponding to AM allocating a container and AM 
> releasing a container
> We can have different log levels even for NM and RM audit logs and move these 
> successful container related logs to DEBUG.

This message was sent by Atlassian JIRA

Reply via email to