Just an update, in case anyone's interested in our performance numbers: With the 512MB newSize, our minor GC pauses are generally less than .05s, although we see a fair amount get up around .15s. We still see some promotion failures causing full pauses over a minute occasionally. But we have a script running to automatically restart our regionservers if that happens. Things seem to be going ok right now.
On a related note: If a region server encounters the GC pause of death, will all of the writes in its memstore at the time be lost (without using WAL)? I think it would be. -Sean On Mon, Nov 29, 2010 at 4:49 AM, Friso van Vollenhoven < fvanvollenho...@xebia.com> wrote: > On a slightly related note, we've been running with G1 with default > settings on a 16GB heap for some weeks now. It's never given us trouble, so > I didn't do any real analysis on the GC times, just some eye balling. > > I looked at the longer GCs (everything longer than 1 second: grep -C 5 -i > real=[1-9] gc-hbase.log), which gives a list of full GCs all around 10s. The > minor pauses all appear to be around 0.2s. I can pastebin a GC log if anyone > is interested in the G1 behavior. > > > > Friso > > > > On 29 nov 2010, at 09:47, Ryan Rawson wrote: > > > I'd love to hear the kinds of minor pauses you get... left alone to > > it's devices, 1.6.0_14 or so wants to grow the new gen to 1gb if your > > xmx is large enough, at that size you are looking at 800ms minor > > pauses! > > > > It's a tough subject. > > > > -ryan > > > > On Wed, Nov 24, 2010 at 12:52 PM, Sean Sechrist <ssechr...@gmail.com> > wrote: > >> Interesting. The settings we tried earlier today slowed jobs > significantly, > >> but no failures (yet). We're going to try the 512MB newSize and 60% > >> CMSInitiatingOccupancyFraction. 1 second pauses here and there would be > OK > >> for us.... we just want to avoid the long pauses right now. We'll also > do > >> what we can to avoid swapping. The ganglia metrics on on there. > >> > >> Thanks, > >> Sean > >> > >> On Wed, Nov 24, 2010 at 3:34 PM, Todd Lipcon <t...@cloudera.com> wrote: > >> > >>> On Wed, Nov 24, 2010 at 7:01 AM, Sean Sechrist <ssechr...@gmail.com > >wrote: > >>> > >>>> Hey guys, > >>>> > >>>> I just want to get an idea about how everyone avoids these long GC > pauses > >>>> that cause regionservers to die. > >>>> > >>>> What kind of java heap and garbage collection settings do you use? > >>>> > >>>> What do you do to make sure that the HBase vm never uses swap? I have > >>>> heard > >>>> turning off swap altogether can be dangerous, so right now we have the > >>>> setting vm.swappiness=0. How do you tell if it's using swap? On > Ganglia, > >>>> we > >>>> see the "CPU wio" metric at around 4.5% before one of our crashes. Is > that > >>>> high? > >>>> > >>>> To try to avoid using too much memory, is reducing the memstore > >>>> upper/lower > >>>> limit, or the block cache size a good idea? Should we just tune down > >>>> HBase's > >>>> total heap to try to avoid swap? > >>>> > >>>> In terms of our specific problem: > >>>> > >>>> We seem to keep running into garbage collection pauses that cause the > >>>> regionservers to die. We have mix of some random read jobs, as well as > a > >>>> few > >>>> full-scan jobs (~1.5 billion rows, 800-900GB of data, 1500 regions), > and > >>>> we > >>>> are always inserting data. We would rather sacrifice a little speed > for > >>>> stability, if that means anything. We have 7 nodes (RS + DN + TT) with > >>>> 12GB > >>>> max heap given to HBase, and 24GB memory total. > >>>> > >>>> We were using the following garbage collection options: > >>>> -XX:+UseConcMarkSweepGC -XX:NewSize=64m -XX:MaxNewSize=64m > >>>> -XX:CMSInitiatingOccupancyFraction=75 > >>>> > >>>> After looking at http://wiki.apache.org/hadoop/PerformanceTuning, we > are > >>>> trying to lower NewSize/MaxNewSize to 6m as well as reducing > >>>> CMSInitiatingOccupancyFraction to 50. > >>>> > >>> > >>> Rather than reducing the new size, you should consider increasing new > size > >>> if you're OK with higher latency but fewer long GC pauses. > >>> > >>> GC is a complicated subject, but here are a few rules of thumb: > >>> > >>> - A larger young generation means that the young GC pauses, which are > >>> stop-the-world, will take longer. In my experience it's somewhere > around 1 > >>> second per GB of new size. So, if you're OK with periodic 1 second > pauses, a > >>> large (1GB) new size should be fine. > >>> - A larger young generation also means that less data will get tenured > to > >>> the old generation. This means that the old generation will have to > collect > >>> less often and also that it will become less fragmented. > >>> - In HBase, the long (45second+) pauses generally happen when promotion > >>> fails due to heap fragmentation in the old generation. So, it falls > back to > >>> stop-the-world compacting collection which takes a long time. > >>> > >>> So, in general, a large young gen will reduce the frequency of > super-long > >>> pauses, but will increase the frequency of shorter pauses. > >>> > >>> It sounds like you may be OK with longer young gen pauses, so maybe > >>> consider new size at 512M with your 12G total heap? > >>> > >>> I also wouldn't tune CMSInitiatingOccupancy below 60% - that will cause > CMS > >>> to always be running which isn't that efficient. > >>> > >>> -Todd > >>> > >>> > >>>> > >>>> We see messages like this in our GC logs: > >>>> > >>>>> 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs] > >>>> > >>>> (concurrent mode failure): 10126729K->5760080K(13246464K), > 91.2530340 > >>>>> secs] > >>>> > >>>> > >>>> > >>>> 2010-11-23T14:56:01.383-0500: 61297.449: [GC 61297.449: [ParNew > (promotion > >>>> failed): 57425K->57880K(59008K), 0.1880950 secs]61297.637: > >>>> [CMS2010-11-23T14:56:06.336-0500: 61302.402: [CMS-concurrent-mark: > >>>> 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs] > >>>> (concurrent mode failure): 10126729K->5760080K(13246464K), 91.2530340 > >>>> secs] > >>>> 10181961K->5760080K(13305472K), [CMS Perm : 20252K->20241K(33868K)], > >>>> 91.4413320 secs] [Times: user=24.47 sys=1.07, real=91.44 secs] > >>>> > >>>> There's a lot of questions there, but I definitely appreciate any > advice > >>>> or > >>>> input anybody else has. Thanks so much! > >>>> > >>>> -Sean > >>>> > >>> > >>> > >>> > >>> -- > >>> Todd Lipcon > >>> Software Engineer, Cloudera > >>> > >> > >
