[
https://issues.apache.org/jira/browse/PIG-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915815#action_12915815
]
Yan Zhou commented on PIG-1648:
-------------------------------
Top 5 locations with most data will be used. This has been agreed upon by the
M/R dev.
> Split combination may return too many block locations to map/reduce framework
> -----------------------------------------------------------------------------
>
> Key: PIG-1648
> URL: https://issues.apache.org/jira/browse/PIG-1648
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Yan Zhou
> Assignee: Yan Zhou
> Fix For: 0.8.0
>
>
> For instance, if a small split has block locations h1, h2 and h3; another
> small split has h1, h3, h4. After combination, the composite split contains 4
> block locations. If the number of component splits is big, then the number of
> block locations could be big too. In fact, the number of block locations
> serves as a hint to M/R as the best hosts this composite split should be run
> on so the list should contain a short list, say 5, of the hosts that contain
> the most data in this composite split.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
