[
https://issues.apache.org/jira/browse/MAPREDUCE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-7319:
-----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Log list of mappers at trace level in ShuffleHandler audit log
> --------------------------------------------------------------
>
> Key: MAPREDUCE-7319
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.4.0
> Reporter: Jim Brennan
> Assignee: Jim Brennan
> Priority: Minor
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7319.001.patch
>
>
> [MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which
> is logged at DEBUG level. After enabling it, we found that the list of
> mappers for large jobs was filling up our audit logs. It would be good to
> move the list of mappers to TRACE level to reduce the logging impact without
> disabling the log message entirely.
> For example a log message like this:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit:
> shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers:
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003,
> attempt_1512479762132_1318600_1_00_004393_0_10003,
> attempt_1512479762132_1318600_1_00_005057_0_10003,
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003,
> attempt_1512479762132_1318600_1_00_004058_0_10003,
> attempt_1512479762132_1318600_1_00_004355_0_10003,
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003,
> attempt_1512479762132_1318600_1_00_005174_0_10004,
> attempt_1512479762132_1318600_1_00_003972_0_10002,
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> Would become this with
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit:
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> {noformat}
> And this with
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit:
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> 2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit:
> shuffle for job_1512479762132_1318600 mappers:
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003,
> attempt_1512479762132_1318600_1_00_004393_0_10003,
> attempt_1512479762132_1318600_1_00_005057_0_10003,
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003,
> attempt_1512479762132_1318600_1_00_004058_0_10003,
> attempt_1512479762132_1318600_1_00_004355_0_10003,
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003,
> attempt_1512479762132_1318600_1_00_005174_0_10004,
> attempt_1512479762132_1318600_1_00_003972_0_10002,
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> One question is whether there are any downstream consumers of this audit log
> that might have a problem with this change?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
