[
https://issues.apache.org/jira/browse/MAPREDUCE-801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737292#action_12737292
]
Arun C Murthy commented on MAPREDUCE-801:
-----------------------------------------
bq. Truncating is probably fine. The original proposal was to "discard location
information completely".
Hmm... truncating is probably fine, but a couple of points to ponder:
# The #locations per split to keep should probably be a cluster-wide config
limit?
# Should we pick first n locations or pick randomly? I'd lean towards randomly
picked splits in light of features such as HADOOP-548
# We should do truncation on both the JobClient _and_ JobTracker to be wary of
DOS if a malicious client submits too many locations per split...
Thoughts?
> MAPREDUCE framework should issue warning with too many locations for a split
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-801
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-801
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Hong Tang
>
> Customized input-format may be buggy and report misleading locations through
> input-split, an example of which is PIG-878. When an input split returns too
> many locations, it would not only artificially inflate the percentage of data
> local or rack local maps, but also force scheduler to use more memory and
> work harder to conduct task assignment.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
