[
https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321305#comment-15321305
]
Jason Lowe commented on MAPREDUCE-6690:
---------------------------------------
Implementing the check in MapReduce allows for fast-failure and more
accurate/informative errors to the client. The check in MapReduce can prevent
an unnecessary upload of one or more resources to the staging area in HDFS
because the client knows the job is going to fail anyway. Also YARN-5192 will
only be able to detect the error when a container starts to localize on a node
that asks for a resource set that violate the limits. Since MapReduce
localizes everything for all containers (including the AM) it will fail under
YARN-5192 as soon as the AM tries to run on a node, but it might take a while
for the AM to get scheduled. As for error reporting, if the violation comes
from one or more files that were submitted locally then the paths via a
YARN-5192 check will be for HDFS staging directories rather than the local path
the client originally specified. The error also will not be reported to the
job client submitting the job unless it hangs around to monitor the job after
submission. With this check the job client will get the error directly when it
tries to submit.
If we don't care much about these differences then we can just go with the
YARN-5192 implementation.
> Limit the number of resources a single map reduce job can submit for
> localization
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6690
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Attachments: MAPREDUCE-6690-trunk-v1.patch,
> MAPREDUCE-6690-trunk-v2.patch, MAPREDUCE-6690-trunk-v3.patch
>
>
> Users will sometimes submit a large amount of resources to be localized as
> part of a single map reduce job. This can cause issues with YARN localization
> that destabilize the cluster and potentially impact other user jobs. These
> resources are specified via the files, libjars, archives and jobjar command
> line arguments or directly through the configuration (i.e. distributed cache
> api). The resources specified could be too large in multiple dimensions:
> # Total size
> # Number of files
> # Size of an individual resource (i.e. a large fat jar)
> We would like to encourage good behavior on the client side by having the
> option of enforcing resource limits along the above dimensions.
> There should be a separate effort to enforce limits at the YARN layer on the
> server side, but this jira is only covering the map reduce layer on the
> client side. In practice, having these client side limits will get us a long
> way towards preventing these localization anti-patterns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
