[
https://issues.apache.org/jira/browse/PIG-2647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265278#comment-14265278
]
Daniel Dai commented on PIG-2647:
---------------------------------
This is nasty and we shall fix immediately. I will commit the patch as is.
A better fix should allow those split combinable in getCombinePigSplits. This
would involve more testing and we shall create a separate Jira to track.
> Split Combining drops splits with empty getLocations()
> ------------------------------------------------------
>
> Key: PIG-2647
> URL: https://issues.apache.org/jira/browse/PIG-2647
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.13.0, 0.14.0, 0.13.1
> Reporter: Alex Levenson
> Assignee: Travis Woodruff
> Attachments: PIG-2647.patch
>
>
> in:
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> which is used by PigInputFormat
> There is an assumption that every split's getLocations() will return a
> non-empty array.
> If the following criteria are met:
> 1) Split combining is turned on
> 2) There is more than one split
> 3) There is at least one split that is smaller than the maxCombineSplitSize
> splits with empty getLocations() will simply be dropped (ignored) without
> warning.
> The hadoop API does not specify that all splits must return a location and
> there are cases where a split may want to return no locations (if the data is
> not in HDFS for example, or if the data is a directory full of HDFS files in
> which case there's not much gained by having locality)
> This is due to the implementation of
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil#getCombinePigSplits
> scans all splits eligible for combining and creates a map of Nodes -> splits,
> then laster iterates through the MAP (not the splits) to do the combining.
> One solution would be to inject a dummy "empty node" into the map.
> Overall the logic in getCombinePigSplits is very complicated and has a lot of
> edge cases, it might be worth cleaning up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
