[jira] [Updated] (MAPREDUCE-7319) Log list of mappers at trace level in ShuffleHandler audit log

2021-02-09 Thread Eric Badger (Jira) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated MAPREDUCE-7319:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Log list of mappers at trace level in ShuffleHandler audit log
> --
>
> Key: MAPREDUCE-7319
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7319.001.patch
>
>
> [MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which 
> is logged at DEBUG level.  After enabling it, we found that the list of 
> mappers for large jobs was filling up our audit logs.  It would be good to 
> move the list of mappers to TRACE level to reduce the logging impact without 
> disabling the log message entirely.
> For example a log message like this:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> Would become this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> {noformat}
> And this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> 2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> One question is whether there are any downstream consumers of this audit log 
> that might have a problem with this change?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7319) Log list of mappers at trace level in ShuffleHandler audit log

2021-02-09 Thread Eric Badger (Jira) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated MAPREDUCE-7319:
---
Fix Version/s: 3.2.3
   2.10.2
   3.3.1
   3.1.5
   3.4.0

+1

Thanks for the patch, [~Jim_Brennan]! I've committed this to trunk (3.4), 
branch-3.3, branch-3.2, branch-3.1, and branch-2.10

> Log list of mappers at trace level in ShuffleHandler audit log
> --
>
> Key: MAPREDUCE-7319
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Attachments: MAPREDUCE-7319.001.patch
>
>
> [MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which 
> is logged at DEBUG level.  After enabling it, we found that the list of 
> mappers for large jobs was filling up our audit logs.  It would be good to 
> move the list of mappers to TRACE level to reduce the logging impact without 
> disabling the log message entirely.
> For example a log message like this:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> Would become this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> {noformat}
> And this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> 2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> One question is whether there are any downstream consumers of this audit log 
> that might have a problem with this change?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7319) Log list of mappers at trace level in ShuffleHandler audit log

2021-02-09 Thread Jim Brennan (Jira) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Brennan updated MAPREDUCE-7319:
---
Status: Patch Available  (was: Open)

> Log list of mappers at trace level in ShuffleHandler audit log
> --
>
> Key: MAPREDUCE-7319
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Attachments: MAPREDUCE-7319.001.patch
>
>
> [MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which 
> is logged at DEBUG level.  After enabling it, we found that the list of 
> mappers for large jobs was filling up our audit logs.  It would be good to 
> move the list of mappers to TRACE level to reduce the logging impact without 
> disabling the log message entirely.
> For example a log message like this:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> Would become this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> {noformat}
> And this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> 2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> One question is whether there are any downstream consumers of this audit log 
> that might have a problem with this change?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Updated] (MAPREDUCE-7319) Log list of mappers at trace level in ShuffleHandler audit log

2021-02-08 Thread Jim Brennan (Jira) 


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Brennan updated MAPREDUCE-7319:
---
Attachment: MAPREDUCE-7319.001.patch

> Log list of mappers at trace level in ShuffleHandler audit log
> --
>
> Key: MAPREDUCE-7319
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7319
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Attachments: MAPREDUCE-7319.001.patch
>
>
> [MAPREDUCE-6958] added the content length to ShuffleHandler audit log, which 
> is logged at DEBUG level.  After enabling it, we found that the list of 
> mappers for large jobs was filling up our audit logs.  It would be good to 
> move the list of mappers to TRACE level to reduce the logging impact without 
> disabling the log message entirely.
> For example a log message like this:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> Would become this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=DEBUG}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> {noformat}
> And this with 
> {{log4j.logger.org.apache.hadoop.mapred.ShuffleHandler.audit=TRACE}}:
> {noformat}
> 2018-01-25 23:43:02,669 [New I/O worker #1] DEBUG ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 reducer 241 length 482072
> 2018-01-25 23:43:02,669 [New I/O worker #1] TRACE ShuffleHandler.audit: 
> shuffle for job_1512479762132_1318600 mappers: 
> [attempt_1512479762132_1318600_1_00_004852_0_10003,
> attempt_1512479762132_1318600_1_00_004190_0_10003, 
> attempt_1512479762132_1318600_1_00_004393_0_10003, 
> attempt_1512479762132_1318600_1_00_005057_0_10003, 
> attempt_1512479762132_1318600_1_00_004855_0_10002,
> attempt_1512479762132_1318600_1_00_003976_0_10003, 
> attempt_1512479762132_1318600_1_00_004058_0_10003, 
> attempt_1512479762132_1318600_1_00_004355_0_10003, 
> attempt_1512479762132_1318600_1_00_004436_0_10002,
> attempt_1512479762132_1318600_1_00_004854_0_10003, 
> attempt_1512479762132_1318600_1_00_005174_0_10004, 
> attempt_1512479762132_1318600_1_00_003972_0_10002, 
> attempt_1512479762132_1318600_1_00_004853_0_10002,
> attempt_1512479762132_1318600_1_00_004856_0_10002]
> {noformat}
> One question is whether there are any downstream consumers of this audit log 
> that might have a problem with this change?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org