[
https://issues.apache.org/jira/browse/MAPREDUCE-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809537#comment-13809537
]
Jason Lowe commented on MAPREDUCE-5603:
---------------------------------------
Recently we ran across a jobclient that failed with an OOM error once we
updated the cluster to 0.23.10. The OOM was triggered by the FileInputFormat
listLocatedStatus optimization from MAPREDUCE-1981, as the client now caches
the BlockLocations of all files along with the FileStatus objects it was
caching before. Normally the user can bump the heap size of the client to work
around this issue. However if a job has an input with a particularly large
number of BlockLocations, as this job did, it would be nice if the user had the
option to disable the optimization to reduce the required memory necessary for
input split calculations.
> Ability to disable FileInputFormat listLocatedStatus optimization to save
> client memory
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5603
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5603
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
> Affects Versions: 0.23.10, 2.2.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
>
> It would be nice if users had the option to disable the listLocatedStatus
> optimization in FileInputFormat to save client memory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
