For this, it's more about abstracting object storage behind a S3 API proxy, for which Swift is definitely a candidate. CloudStack 3.0 already has integration with Swift but this is a cleaner integration.
I definitely would like to see what you've done. I wanted to see it in the last OpenStack conference but was unable to make the meeting. Any links where we can take peak? --Alex > -----Original Message----- > From: David Chamard [mailto:[email protected]] > Sent: Wednesday, May 23, 2012 1:15 PM > To: [email protected] > Subject: Re: HDFS and CloudStack? was: [SCM] CloudStack OSS branch s3- > hdfs > > Hi Guys, > > Have you guys considered swift? Rackspace have a 100% availability with > swift in the last 2 years.. Having the meta data included in the object make > it > very scalable + having a database for every container . The key with swift is > a > good infrastructure architecture to get the desired performance.. We > actually built a tab in the Cloudstack UI for swift and an auth system. > > Would you guys be interested with screenshots of the UIs. Our latest version > is becoming stable and very clean. > > Cheers, > > David > > > On Tue, May 22, 2012 at 4:41 PM, Chiradeep Vittal < > [email protected]> wrote: > > > > > On 5/22/12 12:58 PM, "David Nalley" <[email protected]> wrote: > > > > >Hi Chiradeep - > > > > > >I just saw this branch created - which has a fascinating name - sadly > > >I can't find any docs that talk about Hadoop or HDFS on the wiki > > >(there is a parenthetical statement on the wiki that says 'like > > >Hadoop' but nothing much past that) Can you tell us about what your > > >plan is here, objectives? Functional spec exist for any of this work? > > > > > >--David > > > > This is a follow-up to > > http://www.cloudstack.org/blog/121-cloudstack-and-hadoop-a-match- > made- > > in-th > > e-cloud.html > > "Because of its scalability, reliability, performance, and maturity, > > HDFS is a great object store solution for IaaS cloud. We have started > > the development of an S3 API front-end for HDFS" > > > > The S3 API in the 3.0.x branch supports a simple filesystem backing > > store to store objects and MySQL to store metadata such as ACLs and > policies. > > The code is actually written to be plugged in with different backend > > storage systems. > > Really, most of it is implementing the S3BucketAdapter interface. > > However, there's quite a bit of assumptions about using a filesystem. > > > > What I checked into the s3-hdfs branch is a proof-of-concept to plugin > > HDFS as the backing store for objects. > > It still uses MySQL as the metadata store. > > The code isn't very clean yet, and does require a couple of magical > > incantations to configure on the client and the HDFS side. > > I wanted to push something since I will be out-of-office till next week. > > I'll follow up with more detailed documentation then. > > > > I also have a bunch of thoughts on where this could go including > > - scalable metadata storage (no MySQL) > > - using a CloudStack-managed cohort of service vms as S3 API servers > > - testing performance and scalability > > - replacing (or deprecating) the S3 API frontend with a CloudStack > > object store API > > - interaction with the Hadoop community for enhancements to HDFS > > (multi-datacenter storage for example) > > - support for other Object Storage APIs (e.g., Azure, Google Storage, > > all of which are similar) > > > > Again, I'll write these incoherent thoughts more clearly when I get > > back in town. > > > > -- > > Chiradeep > > > >
