[
https://issues.apache.org/jira/browse/HADOOP-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12574060#action_12574060
]
Chris Douglas commented on HADOOP-2919:
---------------------------------------
sort benchmark on 100 nodes:
|| trunk (r632182) || trunk + patch ||
| 31:09 | 25:46 |
> Create fewer copies of buffer data during sort/spill
> ----------------------------------------------------
>
> Key: HADOOP-2919
> URL: https://issues.apache.org/jira/browse/HADOOP-2919
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Attachments: 2919-0.patch
>
>
> Currently, the sort/spill works as follows:
> Let r be the number of partitions
> For each call to collect(K,V) from map:
> * If buffers do not exist, allocate a new DataOutputBuffer to collect K,V
> bytes, allocate r buffers for collecting K,V offsets
> * Write K,V into buffer, noting offsets
> * Register offsets with associated partition buffer, allocating/copying
> accounting buffers if nesc
> * Calculate the total mem usage for buffer and all partition collectors by
> iterating over the collectors
> * If total mem usage is greater than half of io.sort.mb, then start a new
> thread to spill, blocking if another spill is in progress
> For each spill (assuming no combiner):
> * Save references to our K,V byte buffer and accounting data, setting the
> former to null (will be recreated on the next call to collect(K,V))
> * Open a SequenceFile.Writer for this partition
> * Sort each partition separately (the current version of sort reuses, but
> still requires wrapping, indices in IntWritable objects)
> * Build a RawKeyValueIterator of sorted data for the partition
> * Deserialize each key and value and call SequenceFile::append(K,V) on the
> writer for this partition
> There are a number of opportunities for reducing the number of copies,
> creations, and operations we perform in this stage, particularly since
> growing many of the buffers involved requires that we copy the existing data
> to the newly sized allocation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
