[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14125808#comment-14125808 ] Allen Wittenauer commented on YARN-2256: Someone correct me if I'm wrong, but I'm fairly certain that the intent of this information is to be the equivalent of the HDFS audit log. In other words, setting these to debug completely defeats the purpose. Instead, I suspect the real culprit is that the log4j settings are wrong for the node manager process. > Too many nodemanager audit logs are generated > - > > Key: YARN-2256 > URL: https://issues.apache.org/jira/browse/YARN-2256 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.4.0 >Reporter: Varun Saxena > Attachments: YARN-2256.patch > > > Following audit logs are generated too many times(due to the possibility of a > large number of containers) : > 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a > container > 2. In RM - Audit logs corresponding to AM allocating a container and AM > releasing a container > We can have different log levels even for NM and RM audit logs and move these > successful container related logs to DEBUG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062079#comment-14062079 ] Varun Saxena commented on YARN-2256: Only changed log level from INFO to DEBUG. No tests need to be included. Tested the flows manually where the audit log changed appears. > Too many nodemanager audit logs are generated > - > > Key: YARN-2256 > URL: https://issues.apache.org/jira/browse/YARN-2256 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.4.0 >Reporter: Varun Saxena > Attachments: YARN-2256.patch > > > Following audit logs are generated too many times(due to the possibility of a > large number of containers) : > 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a > container > 2. In RM - Audit logs corresponding to AM allocating a container and AM > releasing a container > We can have different log levels even for NM and RM audit logs and move these > successful container related logs to DEBUG. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[
https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062009#comment-14062009
]
Hadoop QA commented on YARN-2256:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655753/YARN-2256.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4305//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4305//console
This message is automatically generated.
> Too many nodemanager audit logs are generated
> -
>
> Key: YARN-2256
> URL: https://issues.apache.org/jira/browse/YARN-2256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, resourcemanager
>Affects Versions: 2.4.0
>Reporter: Varun Saxena
> Attachments: YARN-2256.patch
>
>
> Following audit logs are generated too many times(due to the possibility of a
> large number of containers) :
> 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a
> container
> 2. In RM - Audit logs corresponding to AM allocating a container and AM
> releasing a container
> We can have different log levels even for NM and RM audit logs and move these
> successful container related logs to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[
https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061922#comment-14061922
]
Hadoop QA commented on YARN-2256:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655734/YARN-2256.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4304//console
This message is automatically generated.
> Too many nodemanager audit logs are generated
> -
>
> Key: YARN-2256
> URL: https://issues.apache.org/jira/browse/YARN-2256
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager, resourcemanager
>Affects Versions: 2.4.0
>Reporter: Varun Saxena
> Attachments: YARN-2256.patch
>
>
> Following audit logs are generated too many times(due to the possibility of a
> large number of containers) :
> 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a
> container
> 2. In RM - Audit logs corresponding to AM allocating a container and AM
> releasing a container
> We can have different log levels even for NM and RM audit logs and move these
> successful container related logs to DEBUG.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061908#comment-14061908 ] Varun Saxena commented on YARN-2256: Changed NM container logs to debug level so that they don't appear in audit logs by default. > Too many nodemanager audit logs are generated > - > > Key: YARN-2256 > URL: https://issues.apache.org/jira/browse/YARN-2256 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.4.0 >Reporter: Varun Saxena > > Following audit logs are generated too many times(due to the possibility of a > large number of containers) : > 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a > container > 2. In RM - Audit logs corresponding to AM allocating a container and AM > releasing a container > We can have different log levels even for NM and RM audit logs and move these > successful container related logs to DEBUG. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14061907#comment-14061907 ] Varun Saxena commented on YARN-2256: Adding of log levels in RM and NM is addressed by this issue > Too many nodemanager audit logs are generated > - > > Key: YARN-2256 > URL: https://issues.apache.org/jira/browse/YARN-2256 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.4.0 >Reporter: Varun Saxena > > Following audit logs are generated too many times(due to the possibility of a > large number of containers) : > 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a > container > 2. In RM - Audit logs corresponding to AM allocating a container and AM > releasing a container > We can have different log levels even for NM and RM audit logs and move these > successful container related logs to DEBUG. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2256) Too many nodemanager audit logs are generated
[ https://issues.apache.org/jira/browse/YARN-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054574#comment-14054574 ] Varun Saxena commented on YARN-2256: I will make the following changes : 1. Create new logSuccess and logFailure methods having an additional parameter indicating log level. This can be an enum in RMAuditLogger and NMAuditLogger. 2. The previous logSuccess method in RMAuditLogger will continue printing logs at INFO level. The new method can be used to print logs at appropriate levels. 3. Change the container logs to DEBUG. > Too many nodemanager audit logs are generated > - > > Key: YARN-2256 > URL: https://issues.apache.org/jira/browse/YARN-2256 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.4.0 >Reporter: Varun Saxena > > Following audit logs are generated too many times(due to the possibility of a > large number of containers) : > 1. In NM - Audit logs corresponding to Starting, Stopping and finishing of a > container > 2. In RM - Audit logs corresponding to AM allocating a container and AM > releasing a container > We can have different log levels even for NM and RM audit logs and move these > successful container related logs to DEBUG. -- This message was sent by Atlassian JIRA (v6.2#6252)
