Hi, I was just wondering whether Linux HA (HeartBeat + DRDB) could be an option in this case. What you guys think?
- Imran On Wed, Nov 25, 2009 at 6:29 PM, Murali Krishna. P <muralikpb...@yahoo.com> wrote: > Hi Ryan, > Thanks for the quick response. > > We are planing have this in 2 or 3 data centers for BCP and latency > reasons. Currently application runs in a non-scalable cluster, essentially we > have the data partitioned across multiple fixed columns. The entire cluster > of machines can be considered as m row vs n column setup. The rows ensures > the availability and columns (machines) ensures distribution/partition. As > you see, even if m-1 box goes down in a column, the availability is 100%. (By > availability, I meant all the documents in the cluster are available) So, > suppose we need 20 machines to hold the data, with 40 machines, we are highly > available. Obviosly we have the data in the other data center, but with a > higher latency. We will take the hit only if all of the replicas fails in one > data center. > > > I have a hbase table which created via mapred tool, took almost 1 hour to > load by the loadtable.rb script and to be available for serving. The scale > was 8k regions per server. I am on 0.20.1, r822817 though. I am yet to test > the failure case, but it will take around 1hour/ no.of RS to redistribute? As > you said, i will try this in 0.21. > > I understand that it is difficult to achive availability in a distributed > system, but the problem is solved in hdfs by replicating data across data > nodes. That is why I thought we should do the same thing in hbase with > replicated regions. We need to ensure that the data is never missing. May be > we can afford 1-2 mins not mare than that. I can definitely redirect the > request to the other data center if there is an region unavailable exception, > but has a higher latency as mentioned. > > The muti RS solution will require double the memory which is not a problem > for users who needs region repl > one. Another idea was that the hbase should > create 2 records for each key, like key#1 and key#2 and some how ensure that > they go to different region servers. If key#1 request fails, key#2 can be > used and vise versa. In a steady state, we can round robin to distribute the > load. But the problem here is that , we are replicating the data also, which > will need double the disk space? So, if some how we can replicate the in > region index but not the data, it will be great. > > Thanks, > Murali Krishna > > > > > ________________________________ > From: Ryan Rawson <ryano...@gmail.com> > To: hbase-user@hadoop.apache.org > Sent: Wed, 25 November, 2009 3:25:20 PM > Subject: Re: HBase High Availability > > With multiple masters, the election is mediated by zookeeper and the > idle masters are awaiting the relection cycle. > > The problems with brining regions up after a failure isnt the actual > speed of loading them, but bugs with the master. This is being fixed > in 0.21. It will allow us to much more rapidly bring regions back > online after a failure. > > As for loading a region across multiple servers, this would have to be > thought about quite carefully to see if it is possible. Right now > there is a substantial amount of state loaded that would be changed by > other servers, and you would still have to reload that state anyways. > > We also need to ask ourselves, what does "availability" mean anyways? > For example, if a regionserver failed, does that mean hbase is > offline? The answer would have to be a "no", but certain sections of > data might be offline temporarily. Thus HBase has 100% uptime by this > definition, correct? > > In the annals of distributed computing, you are only protected with > minimal downtime from limited hardware failures. Once you take out > too many nodes, things start failing, that is a given. HBase solves > the data scalability problem, it solves the limited machine failure > problem. > > I highly suggest this presentation: > http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf > > BTW, what is your budget for "near 100% uptime" anyways? How many > datacenters did you plan on using? > > On Wed, Nov 25, 2009 at 1:31 AM, Murali Krishna. P > <muralikpb...@yahoo.com> wrote: >> Hi, >> This is regarding the region unavailability when a region server goes >> down. There will be cases where we have thousands of regions per RS and it >> takes considerable amount of time to redistribute the regions when a node >> fails. The service will be unavailable during that period. I am evaluating >> HBase for an application where we need to guarantee close to 100% >> availability (namenode is still SPOF, leave that). >> >> One simple idea would be to replicate the regions in memory. Can we load >> the same region in multiple region servers? I am not sure about the >> feasibility yet, there will be issues like consistency across these in >> memory replicas. Wanted to know whether there were any thoughts / work >> already going on this area? I saw some related discussion here >> http://osdir.com/ml/hbase-user-hadoop-apache/2009-09/msg00118.html, not sure >> what is the state. >> >> Same needs to be done with the master as well or is it already done with >> ZK? How fast is the master re-election and catalog load currently ? Do we >> always have multiple masters in ready to run state? >> >> >> Thanks, >> Murali Krishna >> > -- Imran M Yousuf Entrepreneur & Software Engineer Smart IT Engineering Dhaka, Bangladesh Email: im...@smartitengineering.com Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557
