Elliot, Can you elaborate on the blocking being a major factor? Are you referring to the default value of 7 slowing down writes? I don't think removing that feature is a great idea. Here are a couple things that it is helpful for:
1.) Slows down the write path so we are less likely to end up with 1000 storefiles per region server 2.) Slows down the write path enough for the end user to realize it is slow, and thus start troubleshooting/optimizing the write path. I think it might be fair to bump to 15 - 20 by default, but this is the LAST option I touch when troubleshooting the write path. Usually when we start to have blocking issues it is caused by a few other issues: 1.) Too many regions for the allowed memstore upper/lower limit and we are flushing too small 2.) Too small of Hlogs/number of HLogs and we are prematurely rolling 3.) Ingest rate is too fast for the memstore size and needs to be raised. 4.) Slow/not enough drives to keep up with the compaction churn and needs to be tuned The defaults for HBase out of the box here are decent for testing, but not optimized for a production workload. If this is not what you were talking about sorry for the long rant :) On Tue, Jul 30, 2013 at 2:34 AM, Elliott Clark <[email protected]> wrote: > G1 doesn't really make our write path much better if you have uneven > region writes (zipfian distribution or the like). > Lately I've been seeing the memstore blocking size per region being a > major factor. In fact I'm thinking of opening a jira to remove it by > default. > > On Mon, Jul 29, 2013 at 4:12 PM, Andrew Purtell <[email protected]> > wrote: > > Having a difficult time avoiding evacuation failures under concurrent > read > > and write stress. Tuning helps but raises contention for CPU. This is > with > > "interesting" heap sizes - 32-128GB. > > > > > > On Mon, Jul 29, 2013 at 3:44 PM, Bryan Beaudreault < > [email protected] > >> wrote: > > > >> Due to the contention for CPU resources? Or why? > >> > >> > >> On Mon, Jul 29, 2013 at 6:23 PM, Andrew Purtell <[email protected]> > >> wrote: > >> > >> > On Mon, Jul 29, 2013 at 3:14 PM, Elliott Clark <[email protected]> > >> wrote: > >> > > >> > > Yes the G1 looks promising if you have a very read heavy workload > >> > > (We've been running it for integration tests for about a month or > so). > >> > > > >> > > >> > That's my experience too. Unfortunately it's less promising once the > >> > workload is mixed. > >> > > >> > -- > >> > Best regards, > >> > > >> > - Andy > >> > > >> > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > >> > (via Tom White) > >> > > >> > > > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > -- Kevin O'Dell Systems Engineer, Cloudera
