[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[ https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822356#comment-13822356 ] Karthik Kambatla commented on YARN-1383: Sorry for the delay, somehow skipped my mind. I remember using the FS event log when debugging at least 4 production issues. On one such occasion, when jobs/tasks were surprisingly slow, I remember using the FS event log to check what nodes the tasks were being assigned to and if those nodes were heartbeating at all. The issue turned out to be node-specific. It might be possible to arrive at the same conclusion through other means, FS event log makes it super-easy to debug these situations. I completely understand it is too expensive to log these messages all the time. However, it would be nice to at least log these at a trace level. When debugging an issue requiring this, we can always turn on trace-level logging. Remove node updates from the Fair Scheduler event log - Key: YARN-1383 URL: https://issues.apache.org/jira/browse/YARN-1383 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1383.patch Writing out a line whenever a node heartbeats is not useful and just too much. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[
https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822945#comment-13822945
]
Karthik Kambatla commented on YARN-1383:
Spoke to Sandy offline.
It should be okay to leave the following debug message and remove the node
updates:
{code}
if (LOG.isDebugEnabled()) {
LOG.debug(nodeUpdate: + nm + cluster capacity: + clusterCapacity);
}
{code}
Remove node updates from the Fair Scheduler event log
-
Key: YARN-1383
URL: https://issues.apache.org/jira/browse/YARN-1383
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.2.0
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: YARN-1383.patch
Writing out a line whenever a node heartbeats is not useful and just too much.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[ https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822210#comment-13822210 ] Lohit Vijayarenu commented on YARN-1383: On big clusters logging for each heartbeat is too much. To debug if NodeManagers are heartbeating, we could possibly use other methods like network connection, stack traces and such. +1 on removing this line. Remove node updates from the Fair Scheduler event log - Key: YARN-1383 URL: https://issues.apache.org/jira/browse/YARN-1383 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1383.patch Writing out a line whenever a node heartbeats is not useful and just too much. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[ https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812649#comment-13812649 ] Tsuyoshi OZAWA commented on YARN-1383: -- How about using LOG.isErrorEnabled/isWarnEnabled/isInfoEnabled instead of introducing new flags for logging if this logging is useful for debugging? Remove node updates from the Fair Scheduler event log - Key: YARN-1383 URL: https://issues.apache.org/jira/browse/YARN-1383 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1383.patch Writing out a line whenever a node heartbeats is not useful and just too much. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[ https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811222#comment-13811222 ] Karthik Kambatla commented on YARN-1383: This heartbeat log was helpful, at least to me, in the past while debugging production issues. Can we not remove it entirely, may be a flag to turn it on/off would be nice. I am not sure if FS event log is always on - if it is, I definitely see merit to turning it off by default and have a flag for users to turn it on when needed. Remove node updates from the Fair Scheduler event log - Key: YARN-1383 URL: https://issues.apache.org/jira/browse/YARN-1383 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1383.patch Writing out a line whenever a node heartbeats is not useful and just too much. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1383) Remove node updates from the Fair Scheduler event log
[ https://issues.apache.org/jira/browse/YARN-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811413#comment-13811413 ] Sandy Ryza commented on YARN-1383: -- Do you remember in what way the node updates were helpful? I'm leery of introducing another configuration option for something that is so rarely used. Remove node updates from the Fair Scheduler event log - Key: YARN-1383 URL: https://issues.apache.org/jira/browse/YARN-1383 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: YARN-1383.patch Writing out a line whenever a node heartbeats is not useful and just too much. -- This message was sent by Atlassian JIRA (v6.1#6144)
