[jira] [Commented] (YARN-1291) RM INFO logs limit scheduling speed
[ https://issues.apache.org/jira/browse/YARN-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076059#comment-14076059 ] Varun Saxena commented on YARN-1291: Hi [~sandyr], I had raised YARN-2287 which is also about printing of too many RM Audit logs in critical flow. For this, in the patch, I had added support for printing audit logs at different log levels and changed container logs in RM and NM to DEBUG. I didnt remove the audit logs as I wasnt sure if these audit logs are really required or not. RM INFO logs limit scheduling speed --- Key: YARN-1291 URL: https://issues.apache.org/jira/browse/YARN-1291 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza I've been running some microbenchmarks to see how fast the Fair Scheduler can fill up a cluster and found its performance is significantly hampered by logging. I tested with 500 (mock) nodes, and found that: * Taking out fair scheduler INFO logs on the critical path brought down the latency from 14000 ms to 6000 ms * Taking out the INFO that RMContainerImpl logs when a container transitions brought it down from 6000 ms to 4000 ms * Taking out RMAuditLogger logs brought it down from 4000 ms to 1700 ms -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1291) RM INFO logs limit scheduling speed
[ https://issues.apache.org/jira/browse/YARN-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13791999#comment-13791999 ] Sandy Ryza commented on YARN-1291: -- I would like to demote the RMContainerImpl state transition log to DEBUG and use an AsyncAppender for the RMAuditLogger (at least make this configurable if not default). [~vinodkv], as these logs are pretty core, wanted to check what your thoughts are on this? RM INFO logs limit scheduling speed --- Key: YARN-1291 URL: https://issues.apache.org/jira/browse/YARN-1291 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza I've been running some microbenchmarks to see how fast the Fair Scheduler can fill up a cluster and found its performance is significantly hampered by logging. I tested with 500 (mock) nodes, and found that: * Taking out fair scheduler INFO logs on the critical path brought down the latency from 14000 ms to 6000 ms * Taking out the INFO that RMContainerImpl logs when a container transitions brought it down from 6000 ms to 4000 ms * Taking out RMAuditLogger logs brought it down from 4000 ms to 1700 ms -- This message was sent by Atlassian JIRA (v6.1#6144)
