All, Sage forgot to mention that of course using these modules requires a few minor changes to Hadoop's configuration classes. If you're comfortable editing their codebase then simply adding the ceph package into their build system will lead you through the process (just fix each build error!), but if you'd rather not go through that the submitted patch is available on their Jira[1] and I've also added the patch and a brief Readme into our repository. Unfortunately it's only available in the unstable branch, so if you want to see it all make sure you check out that branch! Finally, there's a (very brief!) article up on our wiki[2] discussing this as well. Hopefully the Javadoc will provide you all the documentation you need, though. :) -Greg [1]: https://issues.apache.org/jira/browse/HADOOP-6253 [2]: http://ceph.newdream.net/wiki/Hadoop_FileSystem
On Mon, Oct 5, 2009 at 4:41 PM, Sage Weil <s...@newdream.net> wrote: > Hi, > > Greg has spent some time fixing up a Hadoop FileSystem module allowing a > hadoop cluster to use Ceph in place of HDFS. It hasn't seen extensive > testing or benchmarking (we don't use hadoop internally), but it passes > our basic tests and seems to have similar performance to HDFS. > > The main reason Hadoop users might be interested is the scaling problems > people are having with HDFS's namenode. Ceph's MDS maintains minimal > per-inode metadata (no block lists), doesn't require that it all be in > memory, and (perhaps most importantly) has a clustered MDS architecture, > allowing metadata to be spread across tens or possibly hundreds of nodes. > > Anyway, we're very much interested in seeing Ceph perform well for Hadoop. > > The Hadoop module can be found in src/client/hadoop, and has been > submitted for inclusion in the next Hadoop release. It relies on libceph, > which can be built and installed from source, or as a .deb. > > sage ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel
