[
https://issues.apache.org/jira/browse/HADOOP-8705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438560#comment-13438560
]
kapil bhosale commented on HADOOP-8705:
---------------------------------------
How can we use Distributed Cache (Memcached) to store intermediate results
after Map phase, so that those can be used in Reduce phase from Cache.
> Add JSR 107 Caching support
> ----------------------------
>
> Key: HADOOP-8705
> URL: https://issues.apache.org/jira/browse/HADOOP-8705
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Dhruv Kumar
>
> Having a cache on mappers and reducers could be very useful for some use
> cases, including but not limited to:
> 1. Iterative Map Reduce Programs: Some machine learning algorithms frequently
> need access to invariant data (see Mahout) over each iteration of MapReduce
> until convergence. A cache on such nodes could allow easy access to the
> hotset of data without going all the way to the distributed cache.
> 2. Storing of intermediate map and reduce outputs in memory to reduce
> shuffling time. This optimization has been discussed at length in Haloop
> (http://www.ics.uci.edu/~yingyib/papers/HaLoop_camera_ready.pdf).
> There are some other scenarios as well where having a cache could come in
> handy.
> It will be nice to have some sort of pluggable support for JSR 107 compliant
> caches.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
