[
https://issues.apache.org/jira/browse/HADOOP-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12658259#action_12658259
]
Jothi Padmanabhan commented on HADOOP-1338:
-------------------------------------------
Had an offline discussion with Owen and some of the points that came out are:
* Pull several maps from a single host before moving on to the next. When we
reach a stage where we need to stall, we stall (Similar to the existing logic)
* If the map size is big that we would shuffle to disk anyway, we continue
without stalling.
* We need to come up the correct logic for deciding timeouts as it is possible
that other reducers might time out connecting to a TT that is serving a
particular reducer with huge map outputs (since we are pulling several at one
shot now). We will have to ensure that we do not decide erroneously that maps
from a busy TT are faulty.
* It is not possible to have the Map completion events to have map length
information as that would mean having one long per reducer per map. An
alternate idea could be to have the size encoded using say 2 bits per reducer.
Each value could indicate a range - the ranges could be for exampe 0,
1-999,1000-9999, >10000. We need to evaluate if this encoding would be useful
or not and if yes, what the correct ranges should be.
> Improve the shuffle phase by using the "connection: keep-alive" and doing
> batch transfers of files
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1338
> URL: https://issues.apache.org/jira/browse/HADOOP-1338
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
>
> We should do transfers of map outputs at the granularity of
> *total-bytes-transferred* rather than the current way of transferring a
> single file and then closing the connection to the server. A single
> TaskTracker might have a couple of map output files for a given reduce, and
> we should transfer multiple of them (upto a certain total size) in a single
> connection to the TaskTracker. Using HTTP-1.1's keep-alive connection would
> help since it would keep the connection open for more than one file transfer.
> We should limit the transfers to a certain size so that we don't hold up a
> jetty thread indefinitely (and cause timeouts for other clients).
> Overall, this should give us improved performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
