Chris Trezzo created MAPREDUCE-6690:
---------------------------------------
Summary: Limit the number of resources a single map reduce job can
submit for localization
Key: MAPREDUCE-6690
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
Users will sometimes submit a large amount of resources to be localized as part
of a single map reduce job. This can cause issues with YARN localization that
destabilize the cluster and potentially impact other user jobs. These resources
are specified via the files, libjars, archives and jobjar command line
arguments or directly through the configuration (i.e. distributed cache api).
The resources specified could be too large in multiple dimensions:
# Total size
# Number of files
# Size of an individual resource (i.e. a large fat jar)
We would like to encourage good behavior on the client side by having the
option of enforcing resource limits along the above dimensions.
There should be a separate effort to enforce limits at the YARN layer on the
server side, but this jira is only covering the map reduce layer on the client
side. In practice, having these client side limits will get us a long way
towards preventing these localization anti-patterns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
