Yossi, Yeah they will go up because each datanode keeps their MapFile indexes in memory and the regionservers keep a Memcache of max 64MB (configurable, see hbase-default.xml) for each region it owns.
Rule of thumb? Well in hbase-default the maximum a single family can grow inside a single region is 256MB so you can estimate the number of regions you will have, but it also depends on the number of tables and families. For example, if you have a single table with 10 equally filled families, you should expect around 12 regions. Only one family? 120 regions rough. So, based on that number of regions, you can extrapolate the memory needed to host your system. Big nodes with 16GB mem will host way more regions then a EC2 small instance. J-D On Thu, Oct 23, 2008 at 8:22 AM, Yossi Ittach <[EMAIL PROTECTED]> wrote: > Thanks for the quick reply. > > I'm following the jvm Memory consumption (using "top") , and what bothers > me > is that it seems the percentages are just going up and up , and it makes me > kind of worried. > > I'm trying to load the system with 30GB of data (this is a benchmark) . I > estimate that my production environment will require at least 3 times that > size. Is there a rule-of-thumb as to how many region servers I'll need? > > > Vale et me ama > Yossi > > > On Thu, Oct 23, 2008 at 2:14 PM, Jean-Daniel Cryans <[EMAIL PROTECTED] > >wrote: > > > Yossi, > > > > The META region is usually heavily used and it's worst when you use the > web > > UI. Just for the lolz, go on the Master's page (the main page) and hit > > "refresh" a couple of times; you should see that number go high up. > > > > And on how to avoid it, well the only way to split that load would be to > > have the META region do a split but it will require a lot of data hence a > > lot of user regions which I doubt you have on 2 machines. > > > > J-D > > > > On Thu, Oct 23, 2008 at 8:08 AM, Yossi Ittach <[EMAIL PROTECTED]> wrote: > > > > > Hi > > > > > > Using Hbase with 2 Region servers on similar machines , I see that one > > > machine is serving almost 400 requests per second , while the other one > > is > > > serving 0-10 . This cause extreme overload on the first machine. Any > idea > > > what causes it , or how it can be avoided? > > > > > > Thanks! > > > > > > Vale et me ama > > > Yossi > > > > > >
