Alex, Very interesting since we are considering improving the openstack swift s3 middleware. We are also interested and will be testing heavily the cloudstack integration with swift for backend templates/snapshots
https://github.com/cloudops/cs_auth There is the link for the auth system that Will stevens wrote that allows swift to authenticate through the cloudstack api. We can do a webex or teamviewer to show the UI if you are interested. Cheers, David On Wed, May 23, 2012 at 4:21 PM, Alex Huang <[email protected]> wrote: > For this, it's more about abstracting object storage behind a S3 API > proxy, for which Swift is definitely a candidate. CloudStack 3.0 already > has integration with Swift but this is a cleaner integration. > > I definitely would like to see what you've done. I wanted to see it in > the last OpenStack conference but was unable to make the meeting. Any > links where we can take peak? > > --Alex > > > -----Original Message----- > > From: David Chamard [mailto:[email protected]] > > Sent: Wednesday, May 23, 2012 1:15 PM > > To: [email protected] > > Subject: Re: HDFS and CloudStack? was: [SCM] CloudStack OSS branch s3- > > hdfs > > > > Hi Guys, > > > > Have you guys considered swift? Rackspace have a 100% availability with > > swift in the last 2 years.. Having the meta data included in the object > make it > > very scalable + having a database for every container . The key with > swift is a > > good infrastructure architecture to get the desired performance.. We > > actually built a tab in the Cloudstack UI for swift and an auth system. > > > > Would you guys be interested with screenshots of the UIs. Our latest > version > > is becoming stable and very clean. > > > > Cheers, > > > > David > > > > > > On Tue, May 22, 2012 at 4:41 PM, Chiradeep Vittal < > > [email protected]> wrote: > > > > > > > > On 5/22/12 12:58 PM, "David Nalley" <[email protected]> wrote: > > > > > > >Hi Chiradeep - > > > > > > > >I just saw this branch created - which has a fascinating name - sadly > > > >I can't find any docs that talk about Hadoop or HDFS on the wiki > > > >(there is a parenthetical statement on the wiki that says 'like > > > >Hadoop' but nothing much past that) Can you tell us about what your > > > >plan is here, objectives? Functional spec exist for any of this work? > > > > > > > >--David > > > > > > This is a follow-up to > > > http://www.cloudstack.org/blog/121-cloudstack-and-hadoop-a-match- > > made- > > > in-th > > > e-cloud.html > > > "Because of its scalability, reliability, performance, and maturity, > > > HDFS is a great object store solution for IaaS cloud. We have started > > > the development of an S3 API front-end for HDFS" > > > > > > The S3 API in the 3.0.x branch supports a simple filesystem backing > > > store to store objects and MySQL to store metadata such as ACLs and > > policies. > > > The code is actually written to be plugged in with different backend > > > storage systems. > > > Really, most of it is implementing the S3BucketAdapter interface. > > > However, there's quite a bit of assumptions about using a filesystem. > > > > > > What I checked into the s3-hdfs branch is a proof-of-concept to plugin > > > HDFS as the backing store for objects. > > > It still uses MySQL as the metadata store. > > > The code isn't very clean yet, and does require a couple of magical > > > incantations to configure on the client and the HDFS side. > > > I wanted to push something since I will be out-of-office till next > week. > > > I'll follow up with more detailed documentation then. > > > > > > I also have a bunch of thoughts on where this could go including > > > - scalable metadata storage (no MySQL) > > > - using a CloudStack-managed cohort of service vms as S3 API servers > > > - testing performance and scalability > > > - replacing (or deprecating) the S3 API frontend with a CloudStack > > > object store API > > > - interaction with the Hadoop community for enhancements to HDFS > > > (multi-datacenter storage for example) > > > - support for other Object Storage APIs (e.g., Azure, Google Storage, > > > all of which are similar) > > > > > > Again, I'll write these incoherent thoughts more clearly when I get > > > back in town. > > > > > > -- > > > Chiradeep > > > > > > >
