[
https://issues.apache.org/jira/browse/MAPREDUCE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903656#action_12903656
]
Todd Lipcon commented on MAPREDUCE-2038:
----------------------------------------
This is a very interesting direction. We have another use case for HBase bulk
loads, where we know that a given reducer partition is going to end up on a
particular region server (often colocated with a TT). Scheduling the reducer to
run on the same node or rack will ensure a local replica of the HFile when it
comes time to serve it.
Another interesting use case is for aggregation queries where we can make use
of something like a "rack combiner". We can simply implement a Partitioner that
returns the rack index of the mapper, and then schedule that reduce task on the
same rack. Thus we end up with a result set per rack, and can do a second small
job to recombine those. This is not unlike the multilevel query execution trees
used in Dremel - I imagine Hive and Pig's query planners could make use of
plenty of techniques like this.
> Making reduce tasks locality-aware
> ----------------------------------
>
> Key: MAPREDUCE-2038
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2038
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Hong Tang
>
> Currently Hadoop MapReduce framework does not take into consideration of data
> locality when it decides to launch reduce tasks. There are several cases
> where it could become sub-optimal.
> - The map output data for a particular reduce task are not distributed evenly
> across different racks. This could happen when the job does not have many
> maps, or when there is heavy skew in map output data.
> - A reduce task may need to access some side file (e.g. Pig fragmented join,
> or incremental merge of unsorted smaller dataset with an already sorted large
> dataset). It'd be useful to place reduce tasks based on the location of the
> side files they need to access.
> This jira is created for the purpose of soliciting ideas on how we can make
> it better.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
