On Wed, May 12, 2010 at 1:30 PM, Andrew Purtell <apurt...@apache.org> wrote:
> Before recommending Gluster I suggest you set up a test cluster and then > randomly kill bricks. > > Also as pointed out in another mail, you'll want to colocate TaskTrackers > on Gluster bricks to get I/O locality, yet there is no way for Gluster to > export stripe locations back to Hadoop. > > It seems a poor choice. > > - Andy > > > From: Edward Capriolo > > Subject: Re: Using HBase on other file systems > > To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org> > > Date: Wednesday, May 12, 2010, 6:38 AM > > On Tuesday, May 11, 2010, Jeff > > Hammerbacher <ham...@cloudera.com> > > wrote: > > > Hey Edward, > > > > > > I do think that if you compare GoogleFS to HDFS, GFS > > looks more full > > >> featured. > > >> > > > > > > What features are you missing? Multi-writer append was > > explicitly called out > > > by Sean Quinlan as a bad idea, and rolled back. From > > internal conversations > > > with Google engineers, erasure coding of blocks > > suffered a similar fate. > > > Native client access would certainly be nice, but FUSE > > gets you most of the > > > way there. Scalability/availability of the NN, RPC > > QoS, alternative block > > > placement strategies are second-order features which > > didn't exist in GFS > > > until later in its lifecycle of development as well. > > HDFS is following a > > > similar path and has JIRA tickets with active > > discussions. I'd love to hear > > > your feature requests, and I'll be sure to translate > > them into JIRA tickets. > > > > > > I do believe my logic is reasonable. HBase has a lot > > of code designed around > > >> HDFS. We know these tickets that get cited all > > the time, for better random > > >> reads, or for sync() support. HBase gets the > > benefits of HDFS and has to > > >> deal with its drawbacks. Other key value stores > > handle storage directly. > > >> > > > > > > Sync() works and will be in the next release, and its > > absence was simply a > > > result of the youth of the system. Now that that > > limitation has been > > > removed, please point to another place in the code > > where using HDFS rather > > > than the local file system is forcing HBase to make > > compromises. Your > > > initial attempts on this front (caching, HFile, > > compactions) were, I hope, > > > debunked by my previous email. It's also worth noting > > that Cassandra does > > > all three, despite managing its own storage. > > > > > > I'm trying to learn from this exchange and always > > enjoy understanding new > > > systems. Here's what I have so far from your > > arguments: > > > 1) HBase inherits both the advantages and > > disadvantages of HDFS. I clearly > > > agree on the general point; I'm pressing you to name > > some specific > > > disadvantages, in hopes of helping prioritize our > > development of HDFS. So > > > far, you've named things which are either a) not > > actually disadvantages b) > > > no longer true. If you can come up with the > > disadvantages, we'll certainly > > > take them into account. I've certainly got a number of > > them on our roadmap. > > > 2) If you don't want to use HDFS, you won't want to > > use HBase. Also > > > certainly true, but I'm not sure there's not much to > > learn from this > > > assertion. I'd once again ask: why would you not want > > to use HDFS, and what > > > is your choice in its stead? > > > > > > Thanks, > > > Jeff > > > > > > > Jeff, > > > > Let me first mention that you have mentioned some thing as > > fixed, that > > are only fixed in trunk. I consider trunk futureware and I > > do not like > > to have tempral conversations. Even when trunk becomes > > current there > > is no guarentee that the entire problem is solved. After > > all appends > > were fixed in .19 or not , or again? > > > > I rescanned the gfs white paper to support my argument that > > hdfs is > > stripped down. Found > > Writes at offset ARE supported > > Checkpoints > > Application level checkpoints > > Snapshot > > Shadow read only master > > > > hdfs chose features it wanted and ignored others that is > > why I called > > it a pure map reduce implementation. > > > > My main point, is that hbase by nature needs high speed > > random read > > and random write. Hdfs by nature is bad at these things. If > > you can > > not keep a high cache hit rate via large block cache via > > ram hbase is > > going to slam hdfs doing large block reads for small parts > > of files. > > > > So you ask. Me what I would use instead. I do not think > > there is a > > viable alternative in the 100 tb and up range but I do > > think for > > people in the 20 tb range somethink like gluster that is > > very > > performance focused might deliver amazing results in some > > applications. > > > > > > > I did not recommend anything "people in the 20 tb range somethink like gluster that is very performance focused might deliver amazing results in some applications." I used words like "something. like. might." It may just be an interesting avenue of research. And since you mentioned "also as pointed out in another mail, you'll want to colocate TaskTrackers on Gluster bricks to get I/O locality, yet there is no way for Gluster to export stripe locations back to Hadoop." 1) I am sure if someone was so included they could find a way to export that information from Gluster. 2) I think you meant DataNode not TaskTracker. In any case, I remember reading on list that a RegionServer is not guarenteed to be colocated with a datanode, especially after a restart. Someone was going to open a ticket for it.
