Hi Guys, Have you guys considered swift? Rackspace have a 100% availability with swift in the last 2 years.. Having the meta data included in the object make it very scalable + having a database for every container . The key with swift is a good infrastructure architecture to get the desired performance.. We actually built a tab in the Cloudstack UI for swift and an auth system.
Would you guys be interested with screenshots of the UIs. Our latest version is becoming stable and very clean. Cheers, David On Tue, May 22, 2012 at 4:41 PM, Chiradeep Vittal < [email protected]> wrote: > > On 5/22/12 12:58 PM, "David Nalley" <[email protected]> wrote: > > >Hi Chiradeep - > > > >I just saw this branch created - which has a fascinating name - sadly > >I can't find any docs that talk about Hadoop or HDFS on the wiki > >(there is a parenthetical statement on the wiki that says 'like > >Hadoop' but nothing much past that) Can you tell us about what your > >plan is here, objectives? Functional spec exist for any of this work? > > > >--David > > This is a follow-up to > http://www.cloudstack.org/blog/121-cloudstack-and-hadoop-a-match-made-in-th > e-cloud.html > "Because of its scalability, reliability, performance, and maturity, HDFS > is a great object store solution for IaaS cloud. We have started the > development of an S3 API front-end for HDFS" > > The S3 API in the 3.0.x branch supports a simple filesystem backing store > to store objects and MySQL to store metadata such as ACLs and policies. > The code is actually written to be plugged in with different backend > storage systems. > Really, most of it is implementing the S3BucketAdapter interface. However, > there's quite a bit of assumptions about using a filesystem. > > What I checked into the s3-hdfs branch is a proof-of-concept to plugin > HDFS as the backing store for objects. > It still uses MySQL as the metadata store. > The code isn't very clean yet, and does require a couple of magical > incantations to configure on the client and the HDFS side. > I wanted to push something since I will be out-of-office till next week. > I'll follow up with more detailed documentation then. > > I also have a bunch of thoughts on where this could go including > - scalable metadata storage (no MySQL) > - using a CloudStack-managed cohort of service vms as S3 API servers > - testing performance and scalability > - replacing (or deprecating) the S3 API frontend with a CloudStack object > store API > - interaction with the Hadoop community for enhancements to HDFS > (multi-datacenter storage for example) > - support for other Object Storage APIs (e.g., Azure, Google Storage, all > of which are similar) > > Again, I'll write these incoherent thoughts more clearly when I get back > in town. > > -- > Chiradeep > >
