On Tue, May 11, 2010 at 3:51 PM, Jeff Hammerbacher <ham...@cloudera.com>wrote:
> Hey, > > Thanks for the evaluation, Andrew. Ceph certainly is elegant in design; > HDFS, similar to GFS [1], was purpose-built to get into production quickly, > so its current incarnation lacks some of the same elegance. On the other > hand, there are many techniques for making the metadata servers scalable > and > highly available. HDFS has the advantage of already storing hundreds of > petabytes across thousands of organizations, so we're able to guide those > design decisions with empirical data from heavily used clusters. We'd love > to have heavy users of HBase contribute to the discussions of scalability > [2] and availability [3] of HDFS. See also the excellent article from > Konstantin Schvako of Yahoo! on HDFS scalability [4]. > > I've also conducted extensive reviews at both Facebook and now at Cloudera > of alternative file systems, but at this stage, I concur with Andrew: HDFS > is the only reasonable open source choice for production data processing > workloads. I'm also optimistic that the scalability and availability > challenges will be addressed by the (very active and diverse) HDFS > developer > community over the next few years, and we'll benefit from the work that's > already been put into the robustness and manageability of the system. > > Regardless, every technology improves more rapidly when there's strong > competition, so it will be good to see one of these other file systems > emerge as a viable alternative to HDFS for HBase storage some day. > > [1] > > http://cacm.acm.org/magazines/2010/3/76283-gfs-evolution-on-fast-forward/fulltext > [2] https://issues.apache.org/jira/browse/HDFS-1051 > [3] https://issues.apache.org/jira/browse/HDFS-1064 > [4] > > http://developer.yahoo.net/blogs/hadoop/2010/05/scalability_of_the_hadoop_dist.html > > Later, > Jeff > > On Sun, May 9, 2010 at 9:44 AM, Andrew Purtell <apurt...@apache.org> > wrote: > > > Our experience with Gluster 2 is that self heal when a brick drops off > the > > network is very painful. The high performance impact lasts for a long > time. > > I'm not sure but I think Gluster 3 may only rereplicate missing sections > > instead of entire files. On the other hand I would not trust Gluster 3 to > be > > stable (yet). > > > > I've also tried KFS. My experience seem to bear out other observations > that > > it is ~30% slower that HDFS. Also I was unable to keep the chunkservers > up > > on my CentOS 5 based 64 bit systems. I give Sriram shell access so he > could > > poke around coredumps with gdb but there was no satisfactory resolution. > > > > Another team at Trend is looking at Ceph. I think it is a highly > promising > > filesystem but at the moment it is an experimental filesystem undergoing > a > > high rate of development that requires another experimental filesystem > > undergoing a high rate of development (btrfs) for recovery semantics, and > > the web site warns "NOT SAFE YET" or similar. I doubt it has ever been > > tested on clusters > 100 nodes. In contrast, HDFS has been running in > > production on clusters with 1000s of nodes for a long time. > > > > There currently is not a credible competitor to HDFS in my opinion. Ceph > is > > definitely worth keeping an eye on however. I wonder if HDFS will evolve > to > > offer a similar scalable metadata service (NameNode) to compete. > Certainly > > that would improve its scalability and availability story, both issues > today > > presenting barriers to adoption, and barriers for anything layered on > top, > > like HBase. > > > > - Andy > > > > > > > From: Kevin Apte > > > Subject: Using HBase on other file systems > > > To: hbase-user@hadoop.apache.org > > > Date: Sunday, May 9, 2010, 5:08 AM > > > > > > I am wondering if anyone has thought > > > about using HBase on other file systems like "Gluster". I > > > think Gluster may offer much faster performance without > > > exorbitant cost. With Gluster, you would have to > > > fetch the data from the "Storage Bricks" and process it in > > > your own environment. This allows the > > > servers that are used as storage nodes very cheap. > > > > > > > > > > > > > Hbase is the most square peg, round hole piece of software ever (not an insult read on). HDFS was designed for high throughput streaming batch processing. The random access support is not good. hbase gets around the HDFS shortcomings using caching, HFiles, compaction processes, etc to make the HDFS (tape drive) seem great at all these things it is not good at. One compelling reason to use HBase is that you are already using HDFS for other things. IMHO If you do not need HDFS, you do not really need HBASE. One of the other unamed distributed key value stores will get the job done.
