[
https://issues.apache.org/jira/browse/HADOOP-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HADOOP-1719:
--------------------------------
Status: Patch Available (was: Open)
I can give you one simple case where this is going to help. This is how things
will work without the patch:
Assume an app with some maps and reduces, and let's say the map-outputs are
huge. In a given reduce, if in the first iteration of the loop within
{{fetchOutputs}}, we get just one {{map completion}} event, we will schedule
the fetch for that and will not schedule any other fetch, even if some maps are
over, until the first fetch comes back with the fetch result. The time taken by
the first fetch will be in the order of the size of the map-output (at the
least) and we may be leaving unused bandwidth in the network during that
interval. I am taking "first fetch" just to demonstrate the problem in simple
words, but it is true later on also.
With this patch, map-output fetches will be scheduled whenever there are free
fetcher threads and {{map completion}} events are pending.
> Improve the utilization of shuffle copier threads
> -------------------------------------------------
>
> Key: HADOOP-1719
> URL: https://issues.apache.org/jira/browse/HADOOP-1719
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.15.0
>
> Attachments: 1719.1.patch, 1719.patch
>
>
> In the current design, the scheduling of copies is done and the scheduler
> (the main loop in fetchOutputs) won't schedule anything until it hears back
> from at least one of the copier threads. Due to this, the main loop won't
> query the TaskTracker asking for new map locations and may not be using all
> the copiers effectively. This may not be an issue for small-sized map
> outputs, where at steady state, the frequency of such notifications is
> frequent.
> Ideally, we should schedule all what we can, and, depending on how busy we
> currently are, query the tasktracker for more map locations.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
