[
https://issues.apache.org/jira/browse/HADOOP-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465483
]
Sameer Paranjpye commented on HADOOP-248:
-----------------------------------------
Sounds good. How do we deal with map failures? Do we get multiple completion
events when there are re-tries?
> locating map outputs via random probing is inefficient
> ------------------------------------------------------
>
> Key: HADOOP-248
> URL: https://issues.apache.org/jira/browse/HADOOP-248
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.2.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
>
> Currently the ReduceTaskRunner polls the JobTracker for a random list of map
> tasks asking for their output locations. It would be better if the JobTracker
> kept an ordered log and the interface was changed to:
> class MapLocationResults {
> public int getTimestamp();
> public MapOutputLocation[] getLocations();
> }
> interface InterTrackerProtocol {
> ...
> MapLocationResults locateMapOutputs(int prevTimestamp);
> }
> with the intention that each time a ReduceTaskRunner calls locateMapOutputs,
> it passes back the "timestamp" that it got from the previous result. That
> way, reduces can easily find the new MapOutputs. This should help the "ramp
> up" when the maps first start finishing.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
