[
https://issues.apache.org/jira/browse/MAPREDUCE-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13857249#comment-13857249
]
Sandy Ryza commented on MAPREDUCE-5691:
---------------------------------------
In that case, adding network IO as a YARN resource and limiting it using
cgroups might be a way to solve this problem as well.
> Throttle shuffle's bandwidth utilization
> ----------------------------------------
>
> Key: MAPREDUCE-5691
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5691
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.2.0
> Reporter: Liyin Liang
> Attachments: ganglia-slave.jpg
>
>
> In our hadoop cluster, a reducer of a big job can utilize all the bandwidth
> during shuffle phase. Then any task reading data from the machine which
> running that reducer becomes very very slow.
> It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common.
> And create a throttler for Shuffle to throttle each Fetcher.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
