[
https://issues.apache.org/jira/browse/MAPREDUCE-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219159#comment-14219159
]
Hadoop QA commented on MAPREDUCE-5932:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12682585/MAPREDUCE-5932.v04.patch
against trunk revision a9a0cc3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5037//console
This message is automatically generated.
> Provide an option to use a dedicated reduce-side shuffle log
> ------------------------------------------------------------
>
> Key: MAPREDUCE-5932
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5932
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 2.4.0
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5932.v01.patch, MAPREDUCE-5932.v02.patch,
> MAPREDUCE-5932.v03.patch, MAPREDUCE-5932.v04.patch
>
>
> For reducers in large jobs our users cannot easily spot portions of the log
> associated with problems with their code. An example reducer with INFO-level
> logging generates ~3500 lines / ~700KiB lines per second. 95% of the log is
> the client-side of the shuffle {{org.apache.hadoop.mapreduce.task.reduce.*}}
> {code}
> $ wc syslog
> 3642 48192 691013 syslog
> $ grep task.reduce syslog | wc
> 3424 46534 659038
> $ grep task.reduce.ShuffleScheduler syslog | wc
> 1521 17745 251458
> $ grep task.reduce.Fetcher syslog | wc
> 1045 15340 223683
> $ grep task.reduce.InMemoryMapOutput syslog | wc
> 400 4800 72060
> $ grep task.reduce.MergeManagerImpl syslog | wc
> 432 8200 106555
> {code}
> Byte percentage breakdown:
> {code}
> Shuffle total: 95%
> ShuffleScheduler: 36%
> Fetcher: 32%
> InMemoryMapOutput: 10%
> MergeManagerImpl: 15%
> {code}
> While this is information is actually often useful for devops debugging
> shuffle performance issues, the job users are often lost.
> We propose to have a dedicated syslog.shuffle file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
