What JVM is recommended for the new memstore allocator? We swtiched from u23 back to u17 which helped a lot. Is this optimized for a specific JVM or does it not matter?
On Fri, Feb 18, 2011 at 5:46 PM, Todd Lipcon <[email protected]> wrote: > On Fri, Feb 18, 2011 at 12:10 PM, Jean-Daniel Cryans <[email protected] > >wrote: > > > The bigger the heap the longer the GC pause of the world when > > fragmentation requires it, 8GB is "safer". > > > > On my boxes, a stop-the-world on 8G heap is already around 80 seconds... > pretty catastrophic. Of course we've bumped the ZK timeout up to several > minutes these days, but it's just a bandaid. > > > > > > In 0.90.1 you can try enabling the new memstore allocator that seems > > to do a really good job, checkout the jira first: > > https://issues.apache.org/jira/browse/HBASE-3455 > > > > > Yep. Hopefully will have time to do a blog post this weekend about it as > well. In my testing, try as I might, I can't get my region servers to do a > full GC anymore when this is enabled. > > -Todd > > > > On Fri, Feb 18, 2011 at 12:05 PM, Chris Tarnas <[email protected]> wrote: > > > Thank you , ad that bring me to my next question... > > > > > > What is the current recommendation on the max heap size for Hbase if > RAM > > on the server is not an issue? Right now I am at 8GB and have no issues, > can > > I safely do 12GB? The servers have plenty of RAM (48GB) so that should > not > > be an issue - I just want to minimize the risk that GC will cause > problems. > > > > > > thanks again. > > > -chris > > > > > > On Feb 18, 2011, at 11:59 AM, Jean-Daniel Cryans wrote: > > > > > >> That's what I usually recommend, the bigger the flushed files the > > >> better. On the other hand, you only have so much memory to dedicate to > > >> the MemStore... > > >> > > >> J-D > > >> > > >> On Fri, Feb 18, 2011 at 11:50 AM, Chris Tarnas <[email protected]> wrote: > > >>> Would it be a good idea to raise the > hbase.hregion.memstore.flush.size > > if you have really large regions? > > >>> > > >>> -chris > > >>> > > >>> On Feb 18, 2011, at 11:43 AM, Jean-Daniel Cryans wrote: > > >>> > > >>>> Less regions, but it's often a good thing if you have a lot of data > :) > > >>>> > > >>>> It's probably a good thing to bump the HDFS block size to 128 or > 256MB > > >>>> since you know you're going to have huge-ish files. > > >>>> > > >>>> But anyway regarding penalties, I can't think of one that clearly > > >>>> comes out (unless you use a very small heap). The IO usage patterns > > >>>> will change, but unless you flush very small files all the time and > > >>>> need to recompact them into much bigger ones, then it shouldn't > really > > >>>> be an issue. > > >>>> > > >>>> J-D > > >>>> > > >>>> On Fri, Feb 18, 2011 at 11:36 AM, Jason Rutherglen > > >>>> <[email protected]> wrote: > > >>>>>> We are also using a 5Gb region size to keep our region > > >>>>>> counts in the 100-200 range/node per Jonathan Grey's > recommendation. > > >>>>> > > >>>>> So there isn't a penalty incurred from increasing the max region > size > > >>>>> from 256MB to 5GB? > > >>>>> > > >>> > > >>> > > > > > > > > > > > > -- > Todd Lipcon > Software Engineer, Cloudera >
