[
https://issues.apache.org/jira/browse/HADOOP-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568673#action_12568673
]
Amar Kamat commented on HADOOP-910:
-----------------------------------
Here are the results from a fresh run
{noformat}
Number of nodes : 200
Java heap size : 1024mb
io.sort.factor : 10
{noformat}
||#||with patch||trunk||
|1|1hr 4min|1hr 16min|
|2|1hr 4min|1hr 16min|
|3|1hr 6min|1hr 16min|
I observed that this patch performs badly if more tasks are run simultaneously
on a machine. The earlier runs ran 8 tasks simultaneously. In these runs I set
{{mapred.tasktracker.map.tasks.maximum}} and
{{mapred.tasktracker.reduce.tasks.maximum}} to 2.
> Reduces can do merges for the on-disk map output files in parallel with their
> copying
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-910
> URL: https://issues.apache.org/jira/browse/HADOOP-910
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Amar Kamat
> Attachments: HADOOP-910-review.patch
>
>
> Proposal to extend the parallel in-memory-merge/copying, that is being done
> as part of HADOOP-830, to the on-disk files.
> Today, the Reduces dump the map output files to disk and the final merge
> happens only after all the map outputs have been collected. It might make
> sense to parallelize this part. That is, whenever a Reduce has collected
> io.sort.factor number of segments on disk, it initiates a merge of those and
> creates one big segment. If the rate of copying is faster than the merge, we
> can probably have multiple threads doing parallel merges of independent sets
> of io.sort.factor number of segments. If the rate of copying is not as fast
> as merge, we stand to gain a lot - at the end of copying of all the map
> outputs, we will be left with a small number of segments for the final merge
> (which hopefully will feed the reduce directly (via the RawKeyValueIterator)
> without having to hit the disk for writing additional output segments).
> If the disk bandwidth is higher than the network bandwidth, we have a good
> story, I guess, to do such a thing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
