[
https://issues.apache.org/jira/browse/YARN-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280214#comment-15280214
]
Jason Lowe commented on YARN-4882:
----------------------------------
The main motiviation of proposing a separate logger is to allow finer control
in cases where someone wants to see this recovery output but not enable
DEBUG/TRACE for the entire module. We did something similar in the MapReduce
ShuffleHandler so we could audit shuffle transfers without enabling debug for
all ShuffleHandler operations (like verifying HTTP headers, etc.). Admins can
then configure it to work like it does today (i.e.: enable DEBUG/TRACE on the
new logger and leave it going to the same log file as everything else), use a
separate log file for that logger, or leave it disabled.
IIRC recovered completed applications are re-logged to the RM audit logger, so
we may not really need another log at least for the completed case. I'm OK if
we want to lower successful recoveries to the DEBUG/TRACE level even without a
separate logger for them.
> Change the log level to DEBUG for recovering completed applications
> -------------------------------------------------------------------
>
> Key: YARN-4882
> URL: https://issues.apache.org/jira/browse/YARN-4882
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Reporter: Rohith Sharma K S
> Assignee: Daniel Templeton
>
> I think for recovering completed applications no need to log as INFO, rather
> it can be made it as DEBUG. The problem seen from large cluster is if any
> issue happens during RM start up and continuously switching , then RM logs
> are filled with most with recovering applications only.
> There are 6 lines are logged for 1 applications as I shown in below logs,
> then consider RM default value for max-completed applications is 10K. So for
> each switch 10K*6=60K lines will be added which is not useful I feel.
> {noformat}
> 2016-03-01 10:20:59,077 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: Default priority
> level is set to application:application_1456298208485_21507
> 2016-03-01 10:20:59,094 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Recovering
> app: application_1456298208485_21507 with 1 attempts and final state =
> FINISHED
> 2016-03-01 10:20:59,100 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> Recovering attempt: appattempt_1456298208485_21507_000001 with final state:
> FINISHED
> 2016-03-01 10:20:59,107 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1456298208485_21507_000001 State change from NEW to FINISHED
> 2016-03-01 10:20:59,111 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
> application_1456298208485_21507 State change from NEW to FINISHED
> 2016-03-01 10:20:59,112 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=rohith
> OPERATION=Application Finished - Succeeded TARGET=RMAppManager
> RESULT=SUCCESS APPID=application_1456298208485_21507
> {noformat}
> The main problem is missing important information's from the logs before RM
> unstable. Even though log roll back is 50 or 100, in a short period all these
> logs will be rolled out and all the logs contains only RM switching
> information that too recovering applications!!.
> I suggest at least completed applications recovery should be logged as DEBUG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
