[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-12 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202440#comment-16202440
 ] 

Hudson commented on MAPREDUCE-5951:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13078 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13078/])
MAPREDUCE-5951. Add support for the YARN Shared Cache. (ctrezzo: rev 
e46d5bb962b0c942f993afc505b165b1cd96e51b)
* (add) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploaderWithSharedCache.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java
* (add) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/SharedCacheSupport.md
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
* (add) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobResourceUploader.java
* (add) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/SharedCacheConfig.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
* (edit) hadoop-project/src/site/site.xml
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java
* (edit) 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java


> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Fix For: 2.9.0, 3.0.0
>
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-12 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202418#comment-16202418
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Committed to trunk, branch-3.0 and branch-2. Thanks for all the help with 
reviews [~mingma], [~sjlee0], and [~kasha]!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-10 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199540#comment-16199540
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


+1. Thanks for the great work and taking it to completion [~ctrezzo]!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-10 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199008#comment-16199008
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Thank you [~mingma] for the review! I will wait until Thursday to commit in 
case there are any other comments. Otherwise, I plan to commit to trunk, 
branch-3.0 and branch-2.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-09 Thread Ming Ma (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198205#comment-16198205
 ] 

Ming Ma commented on MAPREDUCE-5951:


+1.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, 
> MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, 
> MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v14.patch, 
> MAPREDUCE-5951-trunk-v15.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch, MAPREDUCE-5951-trunk.016.patch, 
> MAPREDUCE-5951-trunk.017.patch, MAPREDUCE-5951-trunk.018.patch, 
> MAPREDUCE-5951-trunk.019.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-06 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194976#comment-16194976
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Thanks for the comment [~mingma]!

bq. Should any code be moved from MR to YARN to make it easier for other YARN 
applications to use shared cache? For example, maybe other applications can 
benefit from part of LocalResourceBuilder or the special care when dealing with 
fragment.

I have thought about this a fair amount. Originally we started pushing more of 
the fragment code down into the YARN layer (see YARN-3637), but later I 
realized that the code dealing with fragments is purely at the MapReduce layer. 
YARN's api does not use fragments. Instead the ContainerLaunchContext expects a 
Map localResources, where the strings are the 
destination file names (i.e. symlinks). We wound up pulling the fragment 
portion back out of YARN (see YARN-7250) because it was not consistent with the 
rest of the YARN api. Additionally, I think that the way MapReduce uses 
fragments right now is very brittle and prone to bugs. Within MapReduce, 
resources with fragments are converted between paths, URIs and URLs multiple 
times throughout the code and each of these three classes supports fragments in 
different ways. If you are not very careful, one could easily drop a fragment.

I also thought about moving LocalResourceBuilder to YARN, but it has a fair 
amount of MapReduce specific things that would need to change. For example:
# All of the parameters are array based due to how MapReduce currently handles 
resources. We could change this, but then that would need additional 
refactoring at the MapReduce level.
# Components from the MapReduce wildcard feature are in this class. We would 
need to figure out if that makes sense at the yarn layer.
# LocalResourceBuilder currently handles fragments, which we would also need to 
figure out if it makes sense at the yarn layer.

At the end of the day, it would not be simply dropping the LocalResourceBuilder 
into YARN and being done. We would have to think about it more. It does seem 
like something YARN could benefit from, along with a resource uploader. I can 
file another jira to cover these topics, but I think it is probably out of 
scope for this jira.

I think in reality the complexity in this jira is due to the way MapReduce 
itself handles resources and the above mentioned issues with fragments. If we 
wanted to implement a generic yarn resource uploader, I think it could be much 
simpler. For example, this is a slightly simplified version of the code devoted 
to using something in the shared cache:
{noformat}
String localPathChecksum = sharedCacheClient.getFileChecksum(localPath);
URL cachedResource = sharedCacheClient.use(appId, localPathChecksum);
LocalResource resource = LocalResource.newInstance(cachedResource,
  LocalResourceType.FILE, LocalResourceVisibility.PUBLIC
  size, timestamp, null, true);
{noformat}

That LocalResource can then be passed directly to the ContainerLaunchContext 
where a symlink can be specified as a String. As you can see, there is no 
innate need for fragments at the YARN layer.

Please let me know if that makes sense or if I have missed something! Thanks.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail:

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-05 Thread Ming Ma (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194044#comment-16194044
 ] 

Ming Ma commented on MAPREDUCE-5951:


Thanks [~ctrezzo]. The code looks good overall. The only question I have at 
this point is if any code should be moved from MR to YARN to make it easier for 
other YARN applications to use shared cache. For example, maybe other 
applications can benefit from part of  LocalResourceBuilder or the special care 
when dealing with fragment.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-05 Thread Andrew Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193832#comment-16193832
 ] 

Andrew Wang commented on MAPREDUCE-5951:


If it's going into 2.9.0, I think it's safe for 3.0.0 too. Please include it in 
branch-3.0 as well, thanks!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-05 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193828#comment-16193828
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Hi [~sjlee0], [~kasha], [~mingma], [~jlowe], and [~vrushalic]!

I have made another push to get MapReduce support for the shared cache 
committed. I have rebased the patch, added documentation and fixed all 
warnings/issues that I see so far. At this point, I need a reviewer for the 
final review. I know this patch is a big one, so if there is anything I can do 
to help with the review process to make it easier to review, or if there is 
someone else who might be interested in the review, please let me know. Some 
good news:

# Much of the patch has already been reviewed by [~kasha] [~sjlee0] and 
[~mingma] during previous iterations.
# I have ensured that the entire feature is behind a switch. As such, when 
disabled (default) there are no effects for the user.
# I have functionally tested this patch on a pseudo distributed cluster.
# I have deployed this patch to a larger test cluster and ran jobs with the 
patch.
# There is very similar code running in production that has been working for 
years at this point.

My main goal is to commit this to trunk and branch-2 (2.9.0). If it can make it 
into branch-3.0 for GA that would be great as well, but I understand that the 
beta is already out ([~andrew.wang] please let me know what you think). Once I 
get a +1 on the patch, I would be happy to do the work to commit.

Thanks in advance for the help and effort. I really do appreciate it!

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-05 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193573#comment-16193573
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

This is the javac warning:
{noformat}
[WARNING] 
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java:[34,44]
 [deprecation] DistributedCache in org.apache.hadoop.mapreduce.filecache has 
been deprecated
{noformat}

LocalResourceBuilder was a class added to fix a checkstyle warning. I have used 
{{@SuppressWarnings("deprecation")}} to silence the warnings around 
DistributedCache usage at the class level. This warning is complaining about 
the import statement. If anyone has an idea for how to apply the annotation to 
the import statement, please let me know. Furthermore, the LocalResourceBuilder 
is simply refactoring the MRApps#parseDistributedCacheArtifacts method, so I do 
not think it makes sense to fix the usage of a deprecated interface in this 
patch, especially since it is used in a lot of places.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-020.patch, MAPREDUCE-5951-trunk-021.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192255#comment-16192255
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
17s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 43s{color} 
| {color:red} root generated 1 new + 1255 unchanged - 16 fixed = 1256 total 
(was 1271) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
 1s{color} | {color:green} root: The patch generated 0 new + 1058 unchanged - 
15 fixed = 1058 total (was 1073) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
11s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
55s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
48s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m  
3s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m 
11s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
41s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 35s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-5951 |
| JIRA Patch URL |

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-04 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191196#comment-16191196
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m  
3s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 14s{color} 
| {color:red} root generated 1 new + 1255 unchanged - 16 fixed = 1256 total 
(was 1271) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
16s{color} | {color:green} root: The patch generated 0 new + 1058 unchanged - 
15 fixed = 1058 total (was 1073) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
2s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
53s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m  
5s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}114m 
18s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 2s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-5951 |
| JIRA Patch URL |

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-10-02 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189232#comment-16189232
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
41s{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client 
generated 0 new + 354 unchanged - 4 fixed = 354 total (was 358) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
42s{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 3 new + 1062 unchanged - 11 fixed = 1065 total (was 1073) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
0s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
44s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 
19s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 29s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestLocalMRNotification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-5951 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12890076/MAPREDUCE-5951-trunk-020.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux f0ce73fc9e8f 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-04-28 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989545#comment-15989545
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

[~xkrogen]

bq. Is this so that the uploading to SCM can be done by the NM, which is a 
privileged user, to have more secure control over it?

Yes exactly. We wanted to ensure that only trusted entities (i.e. the SCM and 
the node manager) were modifying the shared cached directories in HDFS. 
Additionally, we wanted to make sure that the checksum used when adding a 
resource to the cache was computed by a trusted entity as well.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-04-28 Thread Erik Krogen (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989035#comment-15989035
 ] 

Erik Krogen commented on MAPREDUCE-5951:


Ah, excellent point, [~jlowe]... I actually would love to hear the reasoning 
behind the current strategy of  AM downloads 
resource -> AM uploads resource to SCM> rather than the seemingly more 
obvious/simpler . Is this so that the uploading 
to SCM can be done by the NM, which is a privileged user, to have more secure 
control over it?

[~ctrezzo], first off thanks for getting back so quickly! And for the pointer 
to YARN-5727; that's an interesting issue. The public visibility solution is 
certainly simpler from the YARN side and seems pretty reasonable from a point 
of expectation of burden on an application ("you want a publicly shared 
resource? put it somewhere public"). It  doesn't add _too_ much complexity on 
the MR side, though having a separate staging directory just for public 
resources is a bit cumbersome. It also means that other application developers 
will have to build the same type of logic - in general I would lean towards 
more logic pushed into the YARN level so that it is easy for application devs 
to support. I don't have good insight into how difficult your initially 
proposed solution in YARN-5727 would be to implement, though.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-04-28 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988744#comment-15988744
 ] 

Jason Lowe commented on MAPREDUCE-5951:
---

I don't think it really matters whether the jar resource uploaded by the client 
is public or private.  In both cases the HDFS path to which the client posts 
the resource will be removed when the job completes.  If any subsequent jobs 
come along and figure out via the SCM that they can avoid uploading their own, 
redundant copy of the same resource then they will receive a resource path 
within the SCM area which is a _different_ path than the one used by the first 
job.  That means the resource is going to get downloaded to the node again 
because it's in a different location than the first job's resource.

Even if the first job's client uploads the resource to a public directory, no 
other job is going to ask for that resource under the same path.  It will be 
uploaded to a public staging directory which is specific to that app and whose 
path exists only as long as the app.  The problem with having jobs try to share 
resources automatically just from the job client is knowing when the resource 
can be removed, otherwise we could yank it just as another app tries to 
localize it or never clean it up.  That's why the SCM does the necessary ref 
counting to know what's being used and when resources can be freed safely.  If 
we want to avoid the double-download of the resource then the job client will 
need to upload the resource to the SCM directly and then submit the job _after_ 
it has received the public resource path from the SCM.


> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-04-27 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987897#comment-15987897
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Thanks [~xkrogen] for the comment!

bq. is this an oversight, or is this behavior desired?

Originally we just left it private because we wanted to avoid having to change 
the staging directory and that portion of how MapReduce uploaded resources. As 
I am looking more at YARN-5727, I think it makes more sense to do this so that 
the resources are initially uploaded to a public place and explicitly set with 
a public visibility by the MapReduce client. I was thinking of potentially 
adding a public staging directory that is created and cleaned up by the 
MapReduce client along with the current staging directory. [~xkrogen] would you 
have any thoughts on this? [~jlowe] would you have any thoughts on this as well?

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-04-27 Thread Erik Krogen (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987585#comment-15987585
 ] 

Erik Krogen commented on MAPREDUCE-5951:


Hey [~ctrezzo], I have a question about the behavior of this patch. Currently 
the old logic for resource visibility is used, so if a resource is 
world-readable, it will be marked as PUBLIC, else PRIVATE. Given my current 
understanding of this patch's behavior, I see the following scenario:
* Client submits a job with libjar X, which has never been used before. Client 
contacts SCM to mark X as "used", SCM responds that it does not have X.
* Client uploads X to staging directory, which I assume here is _not_ 
world-readable. X is marked as PRIVATE.
* MR-AM localizes X, then uploads it to the shared cache. Other NMs all 
localize X as PRIVATE and do not share it with other applications.
* Client then submits the same job with the same X. Client contacts SCM, and 
SCM responds with a world-readable (755 dirs / 555 file) path inside of the 
shared cache.
* Client does not upload X, and marks X as PUBLIC, since it is currently in a 
world-readable location. 
* MR-AM and NMs all localize X as PUBLIC and share it with other applications.
Please correct me if I am wrong on any of these steps. It seems that it is the 
expected behavior that X is eventually PUBLIC, given that we asked for it to be 
uploaded to the publicly shared cache, but it seems unnecessary for it to be 
marked as PRIVATE the first time around. Do we do this just to avoid changing 
the existing logic for marking a resource as PRIVATE vs PUBLIC, is this an 
oversight, or is this behavior desired?

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-03-29 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948071#comment-15948071
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

This issue is currently waiting on MAPREDUCE-6862 and MAPREDUCE-6846. I would 
like to get those two patches in so I don't have to rebase this multiple times.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-01-19 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830767#comment-15830767
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Note: currently this patch depends on how YARN-3637 is implemented. I will 
adjust this patch once it is committed.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-01-18 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829326#comment-15829326
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

These are the same 3 checkstyle warnings I mentioned in the 
[comment|https://issues.apache.org/jira/browse/MAPREDUCE-5951?focusedCommentId=15755192=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15755192]
 above.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-Overview.001.pdf, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch, MAPREDUCE-5951-trunk.019.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v2.patch, 
> MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
> MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
> MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
> MAPREDUCE-5951-trunk-v9.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-01-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15829318#comment-15829318
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 13s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 50s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s 
{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client 
generated 0 new + 341 unchanged - 4 fixed = 341 total (was 345) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 3 new + 1086 unchanged - 11 fixed = 1089 total (was 1097) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 30s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s 
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 21s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 54s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
34s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 173m 35s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12848192/MAPREDUCE-5951-trunk.019.patch
 |
| JIRA Issue | MAPREDUCE-5951 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 3f6e6dcb73b4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-01-11 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819182#comment-15819182
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Thanks [~sjlee0] for the review! I will work on v19 of the patch to address 
your comments.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-trunk-v1.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
> MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
> MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
> MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2017-01-10 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817206#comment-15817206
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


I went over the patch with Chris in some detail today, and am posting the 
review comments here for the record.

(MRJobConfig.java)
- {{mapreduce.job.jobjar.visibility}} and 
{{mapreduce.job.jobjar.sharedcache.uploadpolicy}} are computed values that are 
not user-facing; in that case we should not even define the defaults so that 
there is no confusion that these values are computed

(JobResourceUploader.java)
- l.159-160: I understand lines 161-165 are there to support programmatic use 
cases of the distributed cache that come in outside of the job submitter code 
path. Can we make the comments clearer so that the intent of this comes 
through? We could also annotate them in {{MRJobConfig}}.
- l.171-172: it might be slightly better to use {{LinkedHashMap}}. That way, 
we'd have a predictable iteration order (the order in which they are specified 
in the user values).
- l.219: To be fair, this is a bug we need to fix, right? Then can we file a 
JIRA and add the JIRA id here?
- l.237-240: I would not worry about handling previous values here. Having 
duplicate paths is not really supported and the worst case scenario here is to 
reset this upload policy with the same value.
- l.260-263: I think we can improve on this, and reconcile the shared cache 
with the wildcard feature. We could see if any resource is uploaded to the 
staging directory, and if so, still preserve the wildcard entry. We also need 
to consider the case where the shared cache is disabled but the wildcard is 
enabled.
- l.291-294: same comment as above
- l.348-351: same comment as above
- l.388: Since we're dealing with a local filesystem URI based on l.381-382, 
the authority check is not meaningful. We should remove this check.
- On a larger note, the path/URI handling between the job jar, libjars, files, 
and archives is not very consistent, which is an existing behavior. We need to 
see if they need to get the same consistent treatment for this to work.

(Job.java)
- l.1446: I realized later that passing an empty map has an effect of nulling 
out the config value; perhaps we could make that more explicit in the javadoc 
and/or comments/code?
- l.1449-1463: nit: it might be slightly easier to read by using a simple 
string concatenation with "+" (JVM internally uses the {{StringBuilder}})
- l.1490: here also it might be better to use {{LinkedHashMap}}


> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-trunk-v1.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
> MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
> MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
> MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2016-12-16 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755575#comment-15755575
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

I have filed MAPREDUCE-6825 and MAPREDUCE-6824 to address the checkstyle issues 
around methods that are too long.

> Add support for the YARN Shared Cache
> -
>
> Key: MAPREDUCE-5951
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
> Project: Hadoop Map/Reduce
>  Issue Type: New Feature
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>  Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-5951-trunk-v1.patch, 
> MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
> MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
> MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
> MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
> MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
> MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
> MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch, 
> MAPREDUCE-5951-trunk.016.patch, MAPREDUCE-5951-trunk.017.patch, 
> MAPREDUCE-5951-trunk.018.patch
>
>
> Implement the necessary changes so that the MapReduce application can 
> leverage the new YARN shared cache (i.e. YARN-1492).
> Specifically, allow per-job configuration so that MapReduce jobs can specify 
> which set of resources they would like to cache (i.e. jobjar, libjars, 
> archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2016-12-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755545#comment-15755545
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 9s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 0s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 3s 
{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client 
generated 0 new + 359 unchanged - 5 fixed = 359 total (was 364) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 3 new + 1086 unchanged - 11 fixed = 1089 total (was 1097) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 33s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s 
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 52s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 54s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 29s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843632/MAPREDUCE-5951-trunk.018.patch
 |
| JIRA Issue | MAPREDUCE-5951 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 618f3b149fde 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2016-12-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750389#comment-15750389
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 51s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 34s 
{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client 
generated 0 new + 359 unchanged - 5 fixed = 359 total (was 364) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 4 new + 1086 unchanged - 11 fixed = 1090 total (was 1097) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s 
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 55s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 24s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
32s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 3s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843339/MAPREDUCE-5951-trunk.017.patch
 |
| JIRA Issue | MAPREDUCE-5951 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  xml  |
| uname | Linux 563b10e197ee 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2016-12-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746848#comment-15746848
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
2s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s 
{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client 
generated 0 new + 359 unchanged - 5 fixed = 359 total (was 364) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s 
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The 
patch generated 48 new + 1086 unchanged - 11 fixed = 1134 total (was 1097) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s 
{color} | {color:red} 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 0s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s 
{color} | {color:green} hadoop-mapreduce-client-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 7s 
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 26s 
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
28s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 19s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
 |
|  |  Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(Configuration,
 Map, LocalResourceType, URI[], long[], long[], boolean[], Map)  At

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-13 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625083#comment-14625083
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


Thanks for the clarification [~ctrezzo].

There is a recent JIRA that got merged which now enables discovery of whether a 
certain container is an AM: YARN-3116. It exposes the container type in the 
container context. Taking advantage of it might simplify some of the work we do 
here.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-13 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14624967#comment-14624967
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Findbugs run was clean this time so I think the previous empty result was just 
wrong. As perviously mentioned the whitespace errors are not part of this patch 
and the style errors are remaining consistent with the current code convention.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-13 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625334#comment-14625334
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


That sounds fine with me. Thanks!

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-13 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625311#comment-14625311
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

[~sjlee0] thanks for pointing out YARN-3116. If there isn't a strong objection, 
for the sake of getting this patch committed I will leverage that work in a 
follow on patch. I can file a separate jira.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-10 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622675#comment-14622675
 ] 

Sangjin Lee commented on MAPREDUCE-5951:


I've seen several cases where the findbugs result from jenkins is empty. Yet, I 
think it'd be a good idea to double check by running {{mvn findbugs:findbugs}} 
at the said project to see if it is really clean.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-10 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622856#comment-14622856
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  18m 33s | Pre-patch trunk has 1 extant 
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 5 new or modified test files. |
| {color:green}+1{color} | javac |   7m 36s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 41s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 57s | The applied patch generated  
13 new checkstyle issues (total was 578, now 589). |
| {color:red}-1{color} | whitespace |   0m 25s | The patch has 7  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   4m 30s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m  4s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 45s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 45s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 108m 54s | Tests passed in 
hadoop-mapreduce-client-jobclient. |
| | | 165m 38s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12744751/MAPREDUCE-5951-trunk-v15.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0824426 |
| Pre-patch Findbugs warnings | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5883/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v15.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-09 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14621524#comment-14621524
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  18m 46s | Findbugs (version 3.0.0) 
appears to be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 5 new or modified test files. |
| {color:green}+1{color} | javac |   7m 51s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 49s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 55s | The applied patch generated  
13 new checkstyle issues (total was 578, now 589). |
| {color:red}-1{color} | whitespace |   0m 25s | The patch has 10  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:red}-1{color} | findbugs |   4m 45s | The patch appears to introduce 1 
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m  4s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 47s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 44s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 108m 35s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 166m 10s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-app |
| Failed unit tests | hadoop.mapred.TestJobSysDirWithDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12744577/MAPREDUCE-5951-trunk-v14.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f4ca530 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/whitespace.txt
 |
| Findbugs warnings | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5879/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v14.patch, MAPREDUCE-5951-trunk-v2.patch, 
 MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
 MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
 MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
 MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-08 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619614#comment-14619614
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Investigating TestLocalJobSubmission test failure.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-06 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616112#comment-14616112
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  19m 16s | Pre-patch trunk has 1 extant 
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 29s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 37s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 59s | The applied patch generated  
13 new checkstyle issues (total was 578, now 589). |
| {color:red}-1{color} | whitespace |   0m 31s | The patch has 8  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   4m 29s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m  4s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 46s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 44s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 108m 34s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 166m 12s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestLocalJobSubmission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12743839/MAPREDUCE-5951-trunk-v13.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 81f3644 |
| Pre-patch Findbugs warnings | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5870/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-07-06 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616148#comment-14616148
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  18m 54s | Pre-patch trunk has 1 extant 
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 37s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 30s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 58s | The applied patch generated  
13 new checkstyle issues (total was 578, now 589). |
| {color:red}-1{color} | whitespace |   0m 21s | The patch has 8  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   4m 23s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m  1s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 45s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 43s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 108m 40s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 165m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestLocalJobSubmission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12743839/MAPREDUCE-5951-trunk-v13.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 81f3644 |
| Pre-patch Findbugs warnings | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5871/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-15 Thread Karthik Kambatla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546352#comment-14546352
 ] 

Karthik Kambatla commented on MAPREDUCE-5951:
-

Nice to see the patch size go down every iteration :)

Looks mostly good. As we discussed offline, Minor comments outside of that:
# Let us fix the use of arrays to capture per-resource upload policies in a 
follow-up JIRA. Can we add a TODO in the source with a JIRA number please.
# Are the test failures related? 
# pom indentation is broken


 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v2.patch, 
 MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
 MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
 MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
 MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-15 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546303#comment-14546303
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

1. All checkstyle errors (except for the unused import) pertain to lines that 
were not modified by this patch.
2. All of the whitespace errors pertain to lines that were not modified by this 
patch.
3. It seems like the unit test run was in a bad state as well. There were a 
bunch of address already in use exceptions on unrelated tests and 
ClassDefNotFoundExceptions on shared cache tests that pass locally.

[~kasha] I can fix the unused import issue, but will wait to post new version 
until I hear from you (just in case you have other comments about the patch). 
Thanks!

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v2.patch, 
 MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
 MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
 MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
 MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-15 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14546261#comment-14546261
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 44s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 37s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 42s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 58s | The applied patch generated  
13 new checkstyle issues (total was 568, now 581). |
| {color:red}-1{color} | whitespace |   0m 21s | The patch has 8  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   3m 52s | The patch does not introduce 
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests |   9m  6s | Tests failed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 45s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 37s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 101m  6s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 153m 31s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf |
|   | hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs |
|   | hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts |
|   | hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempt |
|   | hadoop.mapred.TestMiniMRChildTask |
|   | hadoop.mapred.TestTextOutputFormat |
|   | hadoop.mapred.TestLocalJobSubmission |
|   | hadoop.mapred.TestFileOutputFormat |
|   | hadoop.mapred.lib.TestKeyFieldBasedComparator |
|   | hadoop.mapreduce.TestJobResourceUploader |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12733228/MAPREDUCE-5951-trunk-v12.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 03a293a |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5739/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v2.patch, 
 MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
 MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
 MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
 MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Karthik Kambatla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542053#comment-14542053
 ] 

Karthik Kambatla commented on MAPREDUCE-5951:
-

The latest patch is definitely easier to understand. 

A couple of high-level comments (sorry for missing these earlier):
# We seem to be using arrays to capture upload policies for individual 
resources in JobResourceUploader. I feel that is error-prone, and it would be 
nice to avoid using arrays. 
# Do we need to release the resources? As per our offline discussion, an 
application calling release doesn't really affect when the resources are 
actually cleaned up. May be getting rid of it altogether will help us simplify 
both this patch and the SCM source as well? 

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Jason Lowe (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542840#comment-14542840
 ] 

Jason Lowe commented on MAPREDUCE-5951:
---

bq. MRApps - we are using * instead of job.jar which should work, but I 
wonder if that will be an incompatible behavior change. Jason Lowe, Vinod Kumar 
Vavilapalli - what do you think?

I get why the '*' is desirable since the name of the jar being shared and 
replacing our job jar might not match the job jar name we expect, although I 
think a comment stating why would help explain why we're not using the more 
specific job jar name that normally would be expected.  That would also help 
prevent someone else coming along much later and fixing that.

I think we're probably OK with using * specifically for the job jar because we 
explicitly use mapreduce.job.jar.unpack.pattern when we unarchive it.  That 
means we'll only unarchive the classes and lib portions of the archive and not 
any other files from it, by default.  So by default we shouldn't be picking up 
any other jars within the job jar.  In theory someone might have packed other 
jars into the job jar, modified the unpack pattern property to pick up those 
jars, then explicitly set their classpath to pick up only a portion of those 
jars or all of them in a specific order.  '*' does not guarantee order in any 
way, so that could break that scenario.  I'm not sure that scenario is likely, 
however.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542904#comment-14542904
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

1. [~kasha] I have thought about removing the release api more and also 
discussed with [~sjlee0]. I think it makes sense from a code simplicity 
standpoint to remove the release api. This will eliminate the need for multiple 
arrays and keeping track of the resources you use on the client side. If we 
feel that it is needed later on, we can always add it back in. The major 
consequence of not releasing is that the SCM store will have more resource 
references to keep track of during the cleaner period time (currently 
defaulting to 1 day). For the InMemorySCMStore, this means that there will be 
more SharedCacheResourceReference objects in-memory.

Rough hand-wavy calculation for heapsize over a 24 hour period on a large 
cluster:
* 42k jobs per day x 600 resources per job = 25.2 million resource references
* A resource reference is made up of an ApplicationId and a ShortUserName.
** Let's say the ApplicationId is two longs, so 16 bytes, and the shortUserName 
is 10 characters, so 20 bytes.
** Let's also multiply this number by 3 to account for Object overhead. So (16 
+ 20) * 3 = 108 bytes for a single resource reference.
* 25.2 million * 108 bytes = 2.7 GB of total heap space

2.7 GB of extra memory does not strike me of being too crazy. We can also trade 
off RM load for memory size and run the cleaner at a higher frequency. Thoughts 
from others? If that sounds reasonable, I will file a YARN jira to make the 
change.

2. [~jlowe] I will add a comment that explains why we are now using '*' instead 
of MRJobConfig.JOB_JAR.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543022#comment-14543022
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Note for the above calculation: We can probably get rid of shortUserName from 
the resource reference if we are not doing release calls. If that is the case, 
then the additional memory should be even smaller.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14543038#comment-14543038
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Another note about removal of the release api: Without the release api, long 
running applications will not be able to release resources they are no longer 
using. Their appId will be claiming that resource until the application is 
finished.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-13 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541415#comment-14541415
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 39s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 30s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 36s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 59s | The applied patch generated  
27 new checkstyle issues (total was 568, now 595). |
| {color:red}-1{color} | whitespace |   0m 25s | The patch has 18  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   3m 53s | The patch does not introduce 
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m 36s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 45s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 35s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 103m 52s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 156m 34s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestLocalJobSubmission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12732450/MAPREDUCE-5951-trunk-v11.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2463666 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5720/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-12 Thread Karthik Kambatla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540396#comment-14540396
 ] 

Karthik Kambatla commented on MAPREDUCE-5951:
-

Sorry again for the delay here. Haven't been able to get a large enough chunk 
of time to review this. 

Few comments based on the partial review I have been able to get to so far. 
Will post comments for the remaining review as soon as I can:
# DistributedCache#getPathStringWithoutFragment: Is this to make sure we get 
the path to the file and not its fields? Can we add comments to describe what 
the method is intended to do? And, may be a different name? Can't think of a 
simpler one myself. 
# Job - have some javadoc suggestions, but may be it is simpler to post an 
updated patch once the patch is ready.
# JobImpl - is the cleanup of upload-policies intended to be in 
init-transition? Is that because we don't need the policies once we are done 
uploading the resources? 
# JobResourceUploader
## In isSharedCacheFilesEnabled and co., we should reverse the order of checks. 
isScmAvailable() should come first as it is cheaper. Also, don't need the outer 
parentheses there. 
## Rename getFiles to mergeLocalAndCacheResources and make it static? 

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540464#comment-14540464
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply 
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12705266/MAPREDUCE-5951-trunk-v8.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fe0df59 |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5716/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-12 Thread Karthik Kambatla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541107#comment-14541107
 ] 

Karthik Kambatla commented on MAPREDUCE-5951:
-

Thanks Chris.

Comments on the latest patch:
# DistributedCache changes are all spurious. Omit it from the diff. 
# Remove Job#addArchiveToSharedCacheAndClasspath altogether? 
# JobImpl#cleanupSharedCacheResources should be always called when the job 
finishes, irrespective of success.
# JobResourceUploader
## stopSharedCache should set scClient to null
## We don't need a separate boolean to check if SCM is available. We should be 
able to just use (sClient != null). If we run into any issues talking to SCM, 
we should just abort and call stopSharedCache to avoid using it for the rest of 
the dependencies.
## uploadFiles seems to be trying to handle the case where shared-cache goes 
down after we set all the files that need to be uploaded. We should leave this 
check to the NM. The SCM might come back up by the time the NM tries uploading 
files. Or, it could be available at this point and go down later. 
{code}
  // if scm fails in the middle, we will set shared cache upload policies
  // for all resources
  // to be false. The resources that are shared successfully via
  // SharedCacheClient.use will
  // continued to be shared.
  if (scClient != null  !isScmAvailable()) {
{code}
## uploadFiles checks if shared-cache is available before setting the upload 
policies. I think we should set the upload policies irrespective of SCM's 
availability when shared-cache is not disabled. Also, we should modify the 
below code to have the second if inside the first if.
{code}
  if (isSharedCacheFilesEnabled()) {
newPath = useSharedCache(tmp, conf);
  }

  // need to inform NM to upload the file to shared cache.
  if (newPath == null  isSharedCacheFilesEnabled()) {
filesSCUploadPolicies[indexOfFilesSCUploadPolicies] = true;
  }
{code}
## useSharedCache should check for (scClient != null)
## uploadFiles has a lot of duplication. Can we file a follow-up JIRA to 
simplify it? 
## Also, due to the try-finally in uploadFiles, it is kind of hard to see what 
lines have been changed. Do you think it makes any sense to wrap this in 
another method call, so the indentations don't show spurious changes? I am 
assuming most of this code will be touched by the follow-up JIRA.
## useSharedCache javadoc issue
# MRApps - we are using * instead of job.jar which should work, but I 
wonder if that will be an incompatible behavior change. [~jlowe], [~vinodkv] - 
what do you think? 
# YarnRunner - indentation issue on line 360

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v2.patch, 
 MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, 
 MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, 
 MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, 
 MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541205#comment-14541205
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  14m 35s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 4 new or modified test files. |
| {color:green}+1{color} | javac |   7m 28s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 34s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 59s | The applied patch generated  
30 new checkstyle issues (total was 583, now 612). |
| {color:red}-1{color} | whitespace |   0m 54s | The patch has 23  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   3m 52s | The patch does not introduce 
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m 21s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   0m 46s | Tests passed in 
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests |   1m 36s | Tests passed in 
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 104m 37s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 157m 25s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestLocalJobSubmission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12732403/MAPREDUCE-5951-trunk-v10.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f24452d |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5718/console |


This message was automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-05-12 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540848#comment-14540848
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Added YARN-3637 to address sym-linking of resources at the YARN layer. Will 
remove sym-linking from this patch.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
  Labels: BB2015-05-TBR
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-03-18 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366721#comment-14366721
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12705266/MAPREDUCE-5951-trunk-v8.patch
  against trunk revision 3bc72cc.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5304//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5304//console

This message is automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-03-17 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366656#comment-14366656
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12705251/MAPREDUCE-5951-trunk-v7.patch
  against trunk revision fc90bf7.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

  org.apache.hadoop.mapreduce.TestJobResourceUploader

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5303//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5303//console

This message is automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, 
 MAPREDUCE-5951-trunk-v8.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-02-26 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338912#comment-14338912
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Created  MAPREDUCE-6267 to address comment in 5.1.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-02-12 Thread Karthik Kambatla (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319375#comment-14319375
 ] 

Karthik Kambatla commented on MAPREDUCE-5951:
-

Sorry for the delay in getting to this. Getting a continuous chunk of time to 
look at this somewhat large patch was hard. 

Here are my first round of comments - a combination of high-level and detailed 
comments. Let us see if we can get some of this in through other JIRAs first, 
to allow for a more thorough review.
# DistributedCache changes aren’t central to what this JIRA is trying to 
address. Could we leave them out and address in another JIRA? 
## This has nothing to do with this patch, but it would be nice to make the 
code around setting CLASSPATH_FILES a little more readable. Could we define 
another String prefix to hold “” or classpath, based on whether classpath is 
null. 
# Job
## The new APIs should all be @Unstable
## Let us make the javadoc for the new APIs a little more formal - we don’t 
need to mention SCMClientProtocol.use, or that the APIs are intended for user 
use. Even for the return value, I would go with something like “If shared cache 
is enabled and the resource added successfully, return true. Otherwise, return 
false.”
## How about renaming the methods to addFileToSharedCache, 
addArchiveToSharedCache, addFileToSharedCacheAndClasspath? 
## Make both new methods private static instead of static private.
# JobID changes might not be required. Use ConverterUtils#toApplicationId? 
# JobImpl
## cleanupSharedCacheResources - nit: I would check for (checksums == null || 
checksums.length == 0) and return to save on indentations. 80 chars is already 
too small.
## cleanupSharedCacheUploadPolicies - javadoc should use block comments. Well, 
may be a nit.
# JobSubmitter
## Can we do the code moving from JobSumitter to FileUploader (may be, we need 
a more descriptive name) to another JIRA and look at that first if needed. 
Otherwise, it is hard to review the changes.
## May be, I am misreading the patch. Is this patch hardcoding MR job 
submission to always use SharedCache? If yes, we should definitely avoid that. 
# mapred-default.xml: We need a little more fool-proof config. The way the 
patch currently is, a typo will lead to unexpected behavior without any 
warnings.


 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2015-01-20 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284690#comment-14284690
 ] 

Hadoop QA commented on MAPREDUCE-5951:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12693397/MAPREDUCE-5951-trunk-v6.patch
  against trunk revision dd0228b.

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 13 new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5112//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5112//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5112//console

This message is automatically generated.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, 
 MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, 
 MAPREDUCE-5951-trunk-v6.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache

2014-08-19 Thread Chris Trezzo (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103104#comment-14103104
 ] 

Chris Trezzo commented on MAPREDUCE-5951:
-

Note: #2 is only in the case where shared cache is enabled. There is still no 
behavior change when the shared cache is disabled.

 Add support for the YARN Shared Cache
 -

 Key: MAPREDUCE-5951
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Chris Trezzo
Assignee: Chris Trezzo
 Attachments: MAPREDUCE-5951-trunk-v1.patch, 
 MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch


 Implement the necessary changes so that the MapReduce application can 
 leverage the new YARN shared cache (i.e. YARN-1492).
 Specifically, allow per-job configuration so that MapReduce jobs can specify 
 which set of resources they would like to cache (i.e. jobjar, libjars, 
 archives, files).



--
This message was sent by Atlassian JIRA
(v6.2#6252)