[
https://issues.apache.org/jira/browse/MAPREDUCE-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13858531#comment-13858531
]
Liyin Liang commented on MAPREDUCE-5691:
----------------------------------------
[~sandyr] as a long-term work, limiting network IO using cgroups is a better
way to solve this problem.
[~jira.shegalov] Our cluster users run thousands of jobs every day. Its
difficult for them to set parameters for specific job.
> Throttle shuffle's bandwidth utilization
> ----------------------------------------
>
> Key: MAPREDUCE-5691
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5691
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.2.0
> Reporter: Liyin Liang
> Attachments: ganglia-slave.jpg
>
>
> In our hadoop cluster, a reducer of a big job can utilize all the bandwidth
> during shuffle phase. Then any task reading data from the machine which
> running that reducer becomes very very slow.
> It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common.
> And create a throttler for Shuffle to throttle each Fetcher.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
