I worked with a cluster of about that size. Once everything is spinning, it requires little attention in my experience. Just have sensible checks (Nagios or alike) on things like disks filling up, especially on the namenode, and have an alert on swap usage (that's usually the beginning of something crashing later on).
Tuning (JVM garbage collection options and HBase configuration) is the hardest part, but possibly your development team does that (or possibly you are the dev team?). A problem can be when the types of workloads that you handle change all the time and you need to re-tune every now and then. Also, you can buy a little peace of mind by having overcapacity. Important part of the work is in development, making sure that the workload is nicely distributed and not hitting a single region server all the time. Important thing is tooling around installation and configuration. So you'll want to be able to do config changes and (rolling) restarts by running a single command / script. Same with adding machines to the cluster. We use cobbler + puppet for that. We managed a 10 node cluster with a 5 person dev/ops/everything team alongside a number of MySQL boxes and some 20-ish other boxes with several purposes. It's pretty doable while still having time to do development. Perhaps Cloudera's SCM express tool can help, but I have never looked at that. Friso On 17 aug. 2011, at 03:27, Sam Seigal wrote: > Hi All, > > I had a question about the operational overhead of maintaining HBase in > production. Would someone care to share their experiences ? We have a team > of 3 DBAs dedicated to maintaining our Oracle cluster. I am curious to know > if we would need the same for HBase. > > I am talking of a small cluster of 7-8 machines handling around 150 million > transactions per hour for the initial rollout. > > What are some of the common operational/maintenance tasks associated > with maintaining a cluster of that size ? How much developer time goes into > this once the cluster is up and running ? > > It would be extremely beneficial to hear some thoughts/experiences. > > Thank you, > > Sam
