Hi All, This email is to draw more attention to YARN-1492 and hopefully gain traction on code reviews. At Twitter we have been running the shared cache on our clusters and it has already served over 36 million requests.
The new shared cache feature is relatively isolated from existing code and is completely enabled/disabled by configuration. When disabled there are no behavioral changes compared to the existing code base. It would be great to see this committed to trunk and even more awesome if it makes 2.6. This is a larger patch, but I have broken it up into a number of sub-tasks in an attempt to make it more digestible for the review process. A couple things to note: 1. The two patches that interact with existing code in a substantial way are: YARN-2236 <https://issues.apache.org/jira/browse/YARN-2236> <https://issues.apache.org/jira/browse/MAPREDUCE-5951> - This patch adds the cache uploader service to the node manager. MAPREDUCE-5951 <https://issues.apache.org/jira/browse/MAPREDUCE-5951> - This patch adds support for the shared cache at the MapReduce layer allowing jobs to cache job jars, lib jars, files and archives. 2. If you would like to try out the entire feature there is a "big bang" patch in YARN-1492 and instructions on how to set it up in a comment on the issue here <https://issues.apache.org/jira/browse/YARN-1492?focusedCommentId=14123617&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14123617> . Please ping me if you have any questions or think there is something else I could do to make reviewing easier. Thanks! Chris Trezzo
