[
https://issues.apache.org/jira/browse/MAPREDUCE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5186:
----------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Rob for the contribution, and thanks Sangjin and Daryn for reviews. I
committed this to trunk and branch-2.
> mapreduce.job.max.split.locations causes some splits created by
> CombineFileInputFormat to fail
> ----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5186
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5186
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Affects Versions: 2.0.4-alpha, 2.2.0
> Reporter: Sangjin Lee
> Assignee: Robert Parker
> Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
> Attachments: MAPREDUCE-5186v1.patch, MAPREDUCE-5186v2.patch,
> MAPREDUCE-5186v3.patch, MAPREDUCE-5186v3.patch
>
>
> CombineFileInputFormat can easily create splits that can come from many
> different locations (during the last pass of creating "global" splits).
> However, we observe that this often runs afoul of the
> mapreduce.job.max.split.locations check that's done by JobSplitWriter.
> The default value for mapreduce.job.max.split.locations is 10, and with any
> decent size cluster, CombineFileInputFormat creates splits that are well
> above this limit.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
