J-D, I have only one table (Huge) , with only 1 family - which means every region has exactly 1 file . Does it mean that I can significantly decrease the size of the MapFile indexes in memory? Also , what do you think will be the impact of increasing the region size (from 256 to 512, for example) , in this scenario?
Vale et me ama Yossi On Thu, Oct 23, 2008 at 2:35 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote: > Yossi, > > Yeah they will go up because each datanode keeps their MapFile indexes in > memory and the regionservers keep a Memcache of max 64MB (configurable, see > hbase-default.xml) for each region it owns. > > Rule of thumb? Well in hbase-default the maximum a single family can grow > inside a single region is 256MB so you can estimate the number of regions > you will have, but it also depends on the number of tables and families. > For > example, if you have a single table with 10 equally filled families, you > should expect around 12 regions. Only one family? 120 regions rough. > > So, based on that number of regions, you can extrapolate the memory needed > to host your system. Big nodes with 16GB mem will host way more regions > then > a EC2 small instance. > > J-D > > On Thu, Oct 23, 2008 at 8:22 AM, Yossi Ittach <[EMAIL PROTECTED]> wrote: > > > Thanks for the quick reply. > > > > I'm following the jvm Memory consumption (using "top") , and what bothers > > me > > is that it seems the percentages are just going up and up , and it makes > me > > kind of worried. > > > > I'm trying to load the system with 30GB of data (this is a benchmark) . I > > estimate that my production environment will require at least 3 times > that > > size. Is there a rule-of-thumb as to how many region servers I'll need? > > > > > > Vale et me ama > > Yossi > > > > > > On Thu, Oct 23, 2008 at 2:14 PM, Jean-Daniel Cryans <[EMAIL PROTECTED] > > >wrote: > > > > > Yossi, > > > > > > The META region is usually heavily used and it's worst when you use the > > web > > > UI. Just for the lolz, go on the Master's page (the main page) and hit > > > "refresh" a couple of times; you should see that number go high up. > > > > > > And on how to avoid it, well the only way to split that load would be > to > > > have the META region do a split but it will require a lot of data hence > a > > > lot of user regions which I doubt you have on 2 machines. > > > > > > J-D > > > > > > On Thu, Oct 23, 2008 at 8:08 AM, Yossi Ittach <[EMAIL PROTECTED]> > wrote: > > > > > > > Hi > > > > > > > > Using Hbase with 2 Region servers on similar machines , I see that > one > > > > machine is serving almost 400 requests per second , while the other > one > > > is > > > > serving 0-10 . This cause extreme overload on the first machine. Any > > idea > > > > what causes it , or how it can be avoided? > > > > > > > > Thanks! > > > > > > > > Vale et me ama > > > > Yossi > > > > > > > > > >
