[
https://issues.apache.org/jira/browse/MAPREDUCE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168588#comment-16168588
]
Jason Lowe commented on MAPREDUCE-6958:
---------------------------------------
Thanks for the review, Chris! I'll update the patch to preserve the existing
format as best as I can.
bq. The shuffle sizes used to be available in the clienttrace log. Was that
removed from the ShuffleHandler at some point?
It does log the content length in the normal logger, but it's on a separate log
line which isn't very friendly given the multithreaded nature of the netty
processing and also not available in the audit log itself.
> Shuffle audit logger should log size of shuffle transfer
> --------------------------------------------------------
>
> Key: MAPREDUCE-6958
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6958
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Attachments: MAPREDUCE-6958.001.patch, MAPREDUCE-6958.002.patch
>
>
> The shuffle audit logger currently logs the job ID and reducer ID but nothing
> about the size of the requested transfer. It calculates this as part of the
> HTTP response headers, so it would be trivial to log the response size. This
> would be very valuable for debugging network traffic storms from the shuffle
> handler.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
