[
https://issues.apache.org/jira/browse/HADOOP-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580559#action_12580559
]
Owen O'Malley commented on HADOOP-2919:
---------------------------------------
Some more comments:
1. rewrite softlimit computations moving the ?: operators down to just
calculate the number of entries & bytes
2. include size of record and size of buffer in map output buffer too small
exception
3. rename inmemuncompressedbytes since it can contain compressed bytes too
4. make combine and spill call close on combiner in finally block
5. remove IllegalArgumentException in memuncompressedbytes declaration
6. make static finals for the index offsets (0, 1, 2, and 3)
> Create fewer copies of buffer data during sort/spill
> ----------------------------------------------------
>
> Key: HADOOP-2919
> URL: https://issues.apache.org/jira/browse/HADOOP-2919
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.17.0
>
> Attachments: 2919-0.patch, 2919-1.patch, 2919-2.patch, 2919-3.patch,
> 2919-4.patch, 2919-5.patch
>
>
> Currently, the sort/spill works as follows:
> Let r be the number of partitions
> For each call to collect(K,V) from map:
> * If buffers do not exist, allocate a new DataOutputBuffer to collect K,V
> bytes, allocate r buffers for collecting K,V offsets
> * Write K,V into buffer, noting offsets
> * Register offsets with associated partition buffer, allocating/copying
> accounting buffers if nesc
> * Calculate the total mem usage for buffer and all partition collectors by
> iterating over the collectors
> * If total mem usage is greater than half of io.sort.mb, then start a new
> thread to spill, blocking if another spill is in progress
> For each spill (assuming no combiner):
> * Save references to our K,V byte buffer and accounting data, setting the
> former to null (will be recreated on the next call to collect(K,V))
> * Open a SequenceFile.Writer for this partition
> * Sort each partition separately (the current version of sort reuses, but
> still requires wrapping, indices in IntWritable objects)
> * Build a RawKeyValueIterator of sorted data for the partition
> * Deserialize each key and value and call SequenceFile::append(K,V) on the
> writer for this partition
> There are a number of opportunities for reducing the number of copies,
> creations, and operations we perform in this stage, particularly since
> growing many of the buffers involved requires that we copy the existing data
> to the newly sized allocation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
