On Wed, Nov 24, 2010 at 7:01 AM, Sean Sechrist <ssechr...@gmail.com> wrote:
> Hey guys, > > I just want to get an idea about how everyone avoids these long GC pauses > that cause regionservers to die. > > What kind of java heap and garbage collection settings do you use? > > What do you do to make sure that the HBase vm never uses swap? I have heard > turning off swap altogether can be dangerous, so right now we have the > setting vm.swappiness=0. How do you tell if it's using swap? On Ganglia, we > see the "CPU wio" metric at around 4.5% before one of our crashes. Is that > high? > > To try to avoid using too much memory, is reducing the memstore upper/lower > limit, or the block cache size a good idea? Should we just tune down > HBase's > total heap to try to avoid swap? > > In terms of our specific problem: > > We seem to keep running into garbage collection pauses that cause the > regionservers to die. We have mix of some random read jobs, as well as a > few > full-scan jobs (~1.5 billion rows, 800-900GB of data, 1500 regions), and we > are always inserting data. We would rather sacrifice a little speed for > stability, if that means anything. We have 7 nodes (RS + DN + TT) with 12GB > max heap given to HBase, and 24GB memory total. > > We were using the following garbage collection options: > -XX:+UseConcMarkSweepGC -XX:NewSize=64m -XX:MaxNewSize=64m > -XX:CMSInitiatingOccupancyFraction=75 > > After looking at http://wiki.apache.org/hadoop/PerformanceTuning, we are > trying to lower NewSize/MaxNewSize to 6m as well as reducing > CMSInitiatingOccupancyFraction to 50. > Rather than reducing the new size, you should consider increasing new size if you're OK with higher latency but fewer long GC pauses. GC is a complicated subject, but here are a few rules of thumb: - A larger young generation means that the young GC pauses, which are stop-the-world, will take longer. In my experience it's somewhere around 1 second per GB of new size. So, if you're OK with periodic 1 second pauses, a large (1GB) new size should be fine. - A larger young generation also means that less data will get tenured to the old generation. This means that the old generation will have to collect less often and also that it will become less fragmented. - In HBase, the long (45second+) pauses generally happen when promotion fails due to heap fragmentation in the old generation. So, it falls back to stop-the-world compacting collection which takes a long time. So, in general, a large young gen will reduce the frequency of super-long pauses, but will increase the frequency of shorter pauses. It sounds like you may be OK with longer young gen pauses, so maybe consider new size at 512M with your 12G total heap? I also wouldn't tune CMSInitiatingOccupancy below 60% - that will cause CMS to always be running which isn't that efficient. -Todd > > We see messages like this in our GC logs: >> >> 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs] > > (concurrent mode failure): 10126729K->5760080K(13246464K), 91.2530340 >> secs] > > > > 2010-11-23T14:56:01.383-0500: 61297.449: [GC 61297.449: [ParNew (promotion > failed): 57425K->57880K(59008K), 0.1880950 secs]61297.637: > [CMS2010-11-23T14:56:06.336-0500: 61302.402: [CMS-concurrent-mark: > 8.844/17.169 secs] [Times: user=75.16 sys=1.34, real=17.17 secs] > (concurrent mode failure): 10126729K->5760080K(13246464K), 91.2530340 > secs] > 10181961K->5760080K(13305472K), [CMS Perm : 20252K->20241K(33868K)], > 91.4413320 secs] [Times: user=24.47 sys=1.07, real=91.44 secs] > > There's a lot of questions there, but I definitely appreciate any advice or > input anybody else has. Thanks so much! > > -Sean > -- Todd Lipcon Software Engineer, Cloudera
