[
https://issues.apache.org/jira/browse/MAPREDUCE-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424864#comment-15424864
]
Hudson commented on MAPREDUCE-6690:
-----------------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10291 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10291/])
MAPREDUCE-6690. Limit the number of resources a single map reduce job (jlowe:
rev f80a7298325a4626638ee24467e2012442e480d4)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* (add)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
> Limit the number of resources a single map reduce job can submit for
> localization
> ---------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6690
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6690
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Chris Trezzo
> Assignee: Chris Trezzo
> Fix For: 2.9.0
>
> Attachments: MAPREDUCE-6690-trunk-v1.patch,
> MAPREDUCE-6690-trunk-v2.patch, MAPREDUCE-6690-trunk-v3.patch,
> MAPREDUCE-6690-trunk-v4.patch, MAPREDUCE-6690-trunk-v5.patch,
> MAPREDUCE-6690-trunk-v6.patch, MAPREDUCE-6690-trunk-v7.patch
>
>
> Users will sometimes submit a large amount of resources to be localized as
> part of a single map reduce job. This can cause issues with YARN localization
> that destabilize the cluster and potentially impact other user jobs. These
> resources are specified via the files, libjars, archives and jobjar command
> line arguments or directly through the configuration (i.e. distributed cache
> api). The resources specified could be too large in multiple dimensions:
> # Total size
> # Number of files
> # Size of an individual resource (i.e. a large fat jar)
> We would like to encourage good behavior on the client side by having the
> option of enforcing resource limits along the above dimensions.
> There should be a separate effort to enforce limits at the YARN layer on the
> server side, but this jira is only covering the map reduce layer on the
> client side. In practice, having these client side limits will get us a long
> way towards preventing these localization anti-patterns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
