[ 
https://issues.apache.org/jira/browse/HADOOP-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557340#action_12557340
 ] 

Doug Cutting commented on HADOOP-2560:
--------------------------------------

> combine multiple input blocks with the same rack into one split [ ... ]

That makes good sense to me.  The new Split class could look a lot like 
MultiFileSplit, but would additionally support a 'getStart(int)' method.  So 
perhaps MultiFileSplit could be extended for this purpose.  FileInputFormat 
could be modified to emit these when the number of splits would otherwise 
exceed some threshold.  But then all subclasses of FileInputFormat would need 
to be modified to be able to accept these.  That wouldn't be too hard.  
FileInputFormat could implement getRecordReader(InputSplit) to break out the 
sub-splits, then call a new method, getRecordReader(FileSplit).  All existing 
subclasses could then just change the signature of their getRecordReader 
implementations in order to support the new feature.


> Combining multiple input blocks into one mapper
> -----------------------------------------------
>
>                 Key: HADOOP-2560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2560
>             Project: Hadoop
>          Issue Type: Bug
>            Reporter: Runping Qi
>
> Currently, an input split contains a consecutive chunk of input file, which 
> by default, corresponding to a DFS block.
> This may lead to a large number of mapper tasks if the input data is large. 
> This leads to the following problems:
> 1. Shuffling cost: since the framework has to move M * R map output segments 
> to the nodes running reducers, 
> larger M means larger shuffling cost.
> 2. High JVM initialization overhead
> 3. Disk fragmentation: larger number of map output files means lower read 
> throughput for accessing them.
> Ideally, you want to keep the number of mappers to no more than 16 times the 
> number of  nodes in the cluster.
> To achive that, we can increase the input split size. However, if a split 
> span over more than one dfs block,
> you lose the data locality scheduling benefits.
> One way to address this problem is to combine multiple input blocks with the 
> same rack into one split.
> If in average we combine B blocks into one split, then we will reduce the 
> number of mappers by a factor of B.
> Since all the blocks for one mapper share a rack, thus we can benefit from 
> rack-aware scheduling.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to