> I presume you intend to run HBase region servers > colocated with HDFS DataNodes.
Yes. --- Seems like we all generally agree that large number of regions per region server may not be the way to go. So coming back to Dhruba's question on having one commit log per region instead of one commit log per region server. Is the number of HDFS files open still a major concern? Is my understanding correct that unavailability window during region server failover is large due to the time it takes to split the shared commit log into a per region log? Instead, if we always had per-region commit logs even in the normal mode of operation, then the unavailability window would be minimized? It does minimize the extent of batch/group commits you can do though-- since you can only batch updates going to the same region. Any other gotchas/issues? regards, Kannan -----Original Message----- From: Andrew Purtell [mailto:apurt...@apache.org] Sent: Tuesday, January 12, 2010 12:50 PM To: hbase-dev@hadoop.apache.org Subject: Re: commit semantics > But would say having a > smaller number of regions per region server (say ~50) be really bad. Not at all. There are some (test) HBase deployments I know of that go pretty vertical, multiple TBs of disk on each node therefore wanting a high number of regions per region server to match that density. That may meet with operational success but it is architecturally suspect. I ran a test cluster once with > 1,000 regions per server on 25 servers, in the 0.19 timeframe. 0.20 is much better in terms of resource demand (less) and liveness (enormously improved), but I still wouldn't recommend it, unless your clients can wait for up to several minutes on blocked reads and writes to affected regions should a node go down. With that many regions per server, it stands to reason just about every client would be affected. The numbers I have for Google's canonical BigTable deployment are several years out of date but they go pretty far in the other direction -- about 100 regions per server is the target. I think it also depends on whether you intend to colocate TaskTrackers with the region servers. I presume you intend to run HBase region servers colocated with HDFS DataNodes. After you have a HBase cluster up for some number of hours, certainly ~24, background compaction will bring the HDFS blocks backing region data local to the server, generally. MapReduce tasks backed by HBase tables will see similar advantages of data locality that you are probably accustomed to with working with files in HDFS. If you mix storage and computation this way it makes sense to seek a balance between the amount of data stored on each node (number of regions being served) and the available computational resources (available CPU cores, time constraints (if any) on task execution). Even if you don't intend to do the above, it's possible that an overly high region density can negatively impact performance if too much I/O load is placed on average on each region server. Adding more servers to spread load would then likely help**. These considerations bias against hosting a very large number of regions per region server. - Andy **: I say likely because this presumes query and edit patterns have been guided as necessary through engineering to be widely distributed in the key space. You have to take some care to avoid hot regions. ----- Original Message ---- > From: Kannan Muthukkaruppan <kan...@facebook.com> > To: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org> > Sent: Tue, January 12, 2010 11:40:00 AM > Subject: RE: commit semantics > > Btw, is there much gains in having a large number of regions-- i.e. to the > tune > of 500 -- per region server? > > I understand that having multiple regions per region server allows finer > grained > rebalancing when new nodes are added or a node goes down. But would say > having a > smaller number of regions per region server (say ~50) be really bad. If a > region > server goes down, 50 other nodes would pick up ~1/50 of its work. Not as good > as > 500 other nodes picking up 1/500 of its work each-- but seems acceptable > still. > Are there other advantages of having a large number of regions per region > server? > > regards, > Kannan > -----Original Message----- > From: jdcry...@gmail.com [mailto:jdcry...@gmail.com] On Behalf Of Jean-Daniel > Cryans > Sent: Tuesday, January 12, 2010 9:42 AM > To: hbase-dev@hadoop.apache.org > Subject: Re: commit semantics > > wrt 1 HLog per region server, this is from the Bigtable paper. Their > main concern is the number of opened files since if you have 1000 > region servers * 500 regions then you may have 100 000 HLogs to > manage. Also you can have more than one file per HLog, so let's say > you have on average 5 log files per HLog that's 500 000 files on HDFS. > > J-D > > On Tue, Jan 12, 2010 at 12:24 AM, Dhruba Borthakur wrote: > > Hi Ryan, > > > > thanks for ur response. > > > >>Right now each regionserver has 1 log, so if 2 puts on different > >>tables hit the same RS, they hit the same HLog. > > > > I understand. My point was that the application could insert the same record > > into two different tables on two different Hbase instances on two different > > piece of hardware. > > > > On a related note, can somebody explain what the tradeoff is if each region > > has its own hlog? are you worried about the number of files in HDFS? or > > maybe the number of sync-threads in the region server? Can multiple hlog > > files provide faster region splits? > > > > > >> I've thought about this issue quite a bit, and I think the sync every > >> 1 rows combined with optional no-sync and low time sync() is the way > >> to go. If you want to discuss this more in person, maybe we can meet > >> up for brews or something. > >> > > > > The group-commit thing I can understand. HDFS does a very similar thing. But > > can you explain your alternative "sync every 1 rows combined with optional > > no-sync and low time sync"? For those applications that have the natural > > characteristics of updating only one row per logical operation, how can they > > be sure that their data has reached some-sort-of-stable-storage unless they > > sync after every row update? > > > > thanks, > > dhruba > >
