Hi there- For what it's worth, although we haven't had this particular issue we've certainly had other bumps and bruises (GC of death, and other metadata issues caused when a split dies during a GC of death, etc.). But there are few general items that helped in stability and performance I thought I'd mention...
1) cluster size I haven't seen any indication of cluster size in this email chain (or a few others that had performance issues), but while 5 is probably the absolute minimum for a cluster (in large part due to HDFS block replication of 3), at larger data volumes things HBase really starts humming at 10 nodes and beyond. Just thought I'd mention it. 2) memory I'm lumping several other threads together but the performance wiki states "don't start HBase for memory". At significant volume this can become an issue (especially if you're still on .89) 3) Cluster restart We schedule a full shutdown and restart of our cluster each week. It's pretty quick, and HBase just seems happier when we do this. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Tuesday, April 12, 2011 1:56 PM To: [email protected] Cc: Robert Gonzalez Subject: Re: HBase is not ready for Primetime On Tue, Apr 12, 2011 at 10:52 AM, Robert Gonzalez <[email protected]> wrote: > Ok, deleted logs that master was complaining about, restarted master > only. Seemed to be stable after a bunch of the messages like the one > below, then restarted regionservers, sans the one that gave me trouble > this morning. Now seems to be up and running again. I don't trust > it, seen this kind of "ok, I'm about to make your life suck" behavior > before. We will see...... > > Understood. Want to post master log after restart for me to check it? Pastebin it and send it here to list or to me privately. St.Ack
