Jim Brennan created MAPREDUCE-7319:
--------------------------------------
Summary: Log list of mappers at trace level in ShuffleHandler
audit log
Key: MAPREDUCE-7319
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: yarn
Affects Versions: 3.4.0
Reporter: Jim Brennan
Assignee: Jim Brennan
[MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which is
logged at DEBUG level. After enabling it, we found that the list of mappers
for large jobs was filling up our audit logs. It would be good to move the
list of mappers to TRACE level to reduce the logging impact without disabling
the log message entirely.
For example a log message like this:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle
for job_1512479762132_1318600 reducer 241 length 482072 mappers:
[attempt_1512479762132_1318600_1_00_004852_0_10003,
attempt_1512479762132_1318600_1_00_004190_0_10003,
attempt_1512479762132_1318600_1_00_004393_0_10003,
attempt_1512479762132_1318600_1_00_005057_0_10003,
attempt_1512479762132_1318600_1_00_004855_0_10002,
attempt_1512479762132_1318600_1_00_003976_0_10003,
attempt_1512479762132_1318600_1_00_004058_0_10003,
attempt_1512479762132_1318600_1_00_004355_0_10003,
attempt_1512479762132_1318600_1_00_004436_0_10002,
attempt_1512479762132_1318600_1_00_004854_0_10003,
attempt_1512479762132_1318600_1_00_005174_0_10004,
attempt_1512479762132_1318600_1_00_003972_0_10002,
attempt_1512479762132_1318600_1_00_004853_0_10002,
attempt_1512479762132_1318600_1_00_004856_0_10002]
{noformat}
Would become this with
{{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle
for job_1512479762132_1318600 reducer 241 length 482072
{noformat}
And this with
{{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
{noformat}
2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: shuffle
for job_1512479762132_1318600 reducer 241 length 482072
2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: shuffle
for job_1512479762132_1318600 mappers:
[attempt_1512479762132_1318600_1_00_004852_0_10003,
attempt_1512479762132_1318600_1_00_004190_0_10003,
attempt_1512479762132_1318600_1_00_004393_0_10003,
attempt_1512479762132_1318600_1_00_005057_0_10003,
attempt_1512479762132_1318600_1_00_004855_0_10002,
attempt_1512479762132_1318600_1_00_003976_0_10003,
attempt_1512479762132_1318600_1_00_004058_0_10003,
attempt_1512479762132_1318600_1_00_004355_0_10003,
attempt_1512479762132_1318600_1_00_004436_0_10002,
attempt_1512479762132_1318600_1_00_004854_0_10003,
attempt_1512479762132_1318600_1_00_005174_0_10004,
attempt_1512479762132_1318600_1_00_003972_0_10002,
attempt_1512479762132_1318600_1_00_004853_0_10002,
attempt_1512479762132_1318600_1_00_004856_0_10002]
{noformat}
One question is whether there are any downstream consumers of this audit log
that might have a problem with this change?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
