[
https://issues.apache.org/jira/browse/HADOOP-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13830371#comment-13830371
]
Mike Liddell commented on HADOOP-10124:
---------------------------------------
patch added.
new flag to govern use of new logic: mapred.submit.shuffle.equalsized.splits.
Default=false.
If flag is true, JobClient will shuffle the splits that share a common size.
> Option to shuffle splits of equal size
> --------------------------------------
>
> Key: HADOOP-10124
> URL: https://issues.apache.org/jira/browse/HADOOP-10124
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Mike Liddell
> Attachments: HADOOP-10124.1.patch
>
>
> Mapreduce split calculation has the following base logic (via JobClient and
> the major InputFormat implementations ):
> ◾enumerate input files in natural (aka linear) order.
> ◾create one split for each 'block-size' of each input. Apart from
> rack-awareness, combining and so on, the input file order remains in its
> natural order.
> ◾sort the splits by size using a stable sort based on splitsize.
> When data from multiple storage services are used in a single hadoop job, we
> get better I/O utilization if the list of splits does round-robin or
> random-access across the services.
> The particular scenario arises in Azure HDInsight where jobs can easily read
> from many storage accounts and each storage account has hard limits on
> throughtput. Concurrent access to the accounts is substantially better than
>
> Two common scenarios can cause non-ideal access pattern:
> 1. many/all input files are the same size
> 2. files have different sizes, but many/all input files have size>blocksize.
> In the second scenario, for each file will have one or more splits with size
> exactly equal to block size so it basically degenerates to the first scenario.
> There are various ways to solve the problem but the simplest is to alter the
> mapreduce JobClient to sort splits by size _and_ randomize the order of
> splits with equal size. This keeps the old behavior effectively unchanged
> while also fixing both common problematic scenarios.
> Some rare scenarios will still suffer bad access patterns due. For example if
> two storage accounts are used and the files from one storage account are all
> smaller than from the other then problems can arise. Addressing these
> scenarios would be further work, perhaps by completely randomizing the split
> order. These problematic scenarios are considered rare and not requiring
> immediate attention.
> If further algorithms for split ordering are necessary, the implementation in
> JobClient will change to being interface-based (eg interface splitOrderer)
> with various standard implementations. At this time there is only the need
> for two implementations and so simple Boolean flag and if/then logic is used.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
