[
https://issues.apache.org/jira/browse/HADOOP-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-4888:
----------------------------------
Attachment: 4888-0.patch
Preliminary patch. There are a few choices that are worth revisiting:
* This adds a HEAD to the fetch, rather than opening a connection and closing
it if there's insufficient space. While this adds a round trip, the connection
will likely be reused. If the impact is measurable, it is possible to restore
the former behavior.
* The hard-coded timeouts and retries remain, but could easily be made
configurable.
I haven't been able to benchmark this. Once that's possible:
* The effect of the fetch buffer size on shuffle time should be quantified
* The number of reduce copier threads should exceed the number of connection
threads so the latter may be reused. A good ratio should be arrived at
experimentally.
* We should measure performance with HttpClient 3.1, too. If the data and S3
support it, we should upgrade.
> Use Apache HttpClient for fetching map outputs
> ----------------------------------------------
>
> Key: HADOOP-4888
> URL: https://issues.apache.org/jira/browse/HADOOP-4888
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Chris Douglas
> Attachments: 4888-0.patch
>
>
> It's worth experimenting with the
> [HttpClient|http://hc.apache.org/httpclient-3.x/] library to speed up the
> shuffle.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
