[
https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated MAPREDUCE-6690:
------------------------------------
Attachment: MAPREDUCE-6690-trunk-v1.patch
Attached is a v1 patch for this jira. A few notes:
# This patch enforces the Total size and Number of files dimensions.
# This patch does not support symlinks. They are broken in other places and it
adds to the complexity of the patch if we want to support them (i.e. handling
symlink cycles). If the community thinks symlink support is a blocker for this
patch, please let me know.
Thanks!
> Limit the number of resources a single map reduce job can submit for
> localization
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6690
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Attachments: MAPREDUCE-6690-trunk-v1.patch
>
>
> Users will sometimes submit a large amount of resources to be localized as
> part of a single map reduce job. This can cause issues with YARN localization
> that destabilize the cluster and potentially impact other user jobs. These
> resources are specified via the files, libjars, archives and jobjar command
> line arguments or directly through the configuration (i.e. distributed cache
> api). The resources specified could be too large in multiple dimensions:
> # Total size
> # Number of files
> # Size of an individual resource (i.e. a large fat jar)
> We would like to encourage good behavior on the client side by having the
> option of enforcing resource limits along the above dimensions.
> There should be a separate effort to enforce limits at the YARN layer on the
> server side, but this jira is only covering the map reduce layer on the
> client side. In practice, having these client side limits will get us a long
> way towards preventing these localization anti-patterns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
