John, this is the most recent work on shared caches for YARN https://issues.apache.org/jira/browse/YARN-1492
it's not yet committed, but will ship later this year -and allow artifacts to be shared by users, using checksums to ensure that the contents are what they say they are On 27 June 2014 16:40, John Lilley <[email protected]> wrote: > Our software doesn't use MapReduce. It is a pure YARN application that is > basically a peer to MapReduce. There are a lot of reasons for this > decision, but the main one is that we have a large code base that already > executes data transformations in a single-server environment, and we wanted > to produce a product without rewriting huge swaths of code. Given that, our > software takes care of many things usually delegated to MapReduce, > including distributed sort/partition (i.e. "the shuffle"). However, > MapReduce has a special place in the ecosystem, in that it creates an > auxiliary service to handle the distribution of shuffle data to reducers. > It doesn't look like third-party apps have an easy time installing aux > services. The JARs for any such service must be in Hadoop's classpath on > all nodes at startup, creating both a management issue and a trust/security > issue. Currently our software places temporary data into HDFS for this > purpose, but we've found that HDFS has a huge overhead in terms of > performance and file handles, even at low replication. We desire to replace > the use of HDFS with a lighter-weight service to manage temp files and > distribute their data. > > Is the slider project something that can address our needs? > > John Lilley > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
