These are a different kind of pause (those caused by blockingStoreFiles). This is HBase stepping in and actually blocking updates to a region because compactions have not been able to keep up with the write load. It could manifest itself in the same way but this is different than shorter pauses caused by periodic offlining of regions during balancing and splits.
Wayne, have you confirmed in your RegionServer logs that the pauses are associated with splits or region movement, and that you are not seeing the blocking store files issue? JG > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Christopher > Tarnas > Sent: Friday, January 14, 2011 7:29 AM > To: [email protected] > Subject: Re: Cluster Wide Pauses > > I have been seeing similar problems and found by raising the > hbase.hregion.memstore.block.multiplier > to above 12 (default is two) and the hbase.hstore.blockingStoreFiles to 16 I > managed to reduce the frequency of the pauses during loads. My nodes are > pretty beefy (48 GB of ram) so I had room to experiment. > > From what I understand that gave the regionservers more buffer before > they had to halt the world to catch up. The pauses still happen but their > impact is less now. > > -chris > > On Fri, Jan 14, 2011 at 8:34 AM, Wayne <[email protected]> wrote: > > > We have not found any smoking gun here. Most likely these are region > > splits on a quickly growing/hot region that all clients get caught waiting > > for. > > > > > > On Thu, Jan 13, 2011 at 7:49 AM, Wayne <[email protected]> wrote: > > > > > Thank you for the lead! We will definitely look closer at the OS logs. > > > > > > > > > On Thu, Jan 13, 2011 at 6:59 AM, Tatsuya Kawano > > ><[email protected] > > >wrote: > > > > > >> > > >> Hi Wayne, > > >> > > >> > We are seeing some TCP Resets on all nodes at the same time, and > > >> sometimes > > >> > quite a lot of them. > > >> > > >> > > >> Have you checked this article from Andrei and Cosmin? They had a > > >> busy firewall to cause network blackout. > > >> > > >> http://hstack.org/hbase-performance-testing/ > > >> > > >> Maybe it's not your case but just for sure. > > >> > > >> Thanks, > > >> > > >> -- > > >> Tatsuya Kawano (Mr.) > > >> Tokyo, Japan > > >> > > >> > > >> On Jan 13, 2011, at 4:52 AM, Wayne <[email protected]> wrote: > > >> > > >> > We are seeing some TCP Resets on all nodes at the same time, and > > >> sometimes > > >> > quite a lot of them. We have yet to correlate the pauses to the > > >> > TCP > > >> resets > > >> > but I am starting to wonder if this is partly a network problem. > > >> > Does Gigabit Ethernet break down on high volume nodes? Do high > > >> > volume nodes > > >> use > > >> > 10G or Infiniband? > > >> > > > >> > > > >> > On Wed, Jan 12, 2011 at 1:52 PM, Stack <[email protected]> wrote: > > >> > > > >> >> Jon asks that you describe your loading in the issue. Would you > > >> >> mind doing so. Ted, stick up in the issue the workload and > > >> >> configs. you are running if you don't mind. I'd like to try it over > > >> >> here. > > >> >> Thanks lads, > > >> >> St.Ack > > >> >> > > >> >> > > >> >> On Wed, Jan 12, 2011 at 9:03 AM, Wayne <[email protected]> > wrote: > > >> >>> Added: https://issues.apache.org/jira/browse/HBASE-3438. > > >> >>> > > >> >>> On Wed, Jan 12, 2011 at 11:40 AM, Wayne <[email protected]> > wrote: > > >> >>> > > >> >>>> We are using 0.89.20100924, r1001068 > > >> >>>> > > >> >>>> We are seeing see it during heavy write load (which is all the > > time), > > >> >> but > > >> >>>> yesterday we had read load as well as write load and saw both > > >> >>>> reads > > >> and > > >> >>>> writes stop for 10+ seconds. The region size is the biggest > > >> >>>> clue we > > >> have > > >> >>>> found from our tests as setting up a new cluster with a 1GB > > >> >>>> max > > >> region > > >> >> size > > >> >>>> and starting to load heavily we will see this a lot for long > > >> >>>> long > > >> time > > >> >>>> frames. Maybe the bigger file gets hung up more easily with a > > split? > > >> >> Your > > >> >>>> description below also fits in that early on the load is not > > balanced > > >> so > > >> >> it > > >> >>>> is easier to stop everything on one node as the balance is not > > great > > >> >> early > > >> >>>> on. I will file a JIRA. I will also try to dig deeper into the > > >> >>>> logs > > >> >> during > > >> >>>> the pauses to find a node that might be stuck in a split. > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> On Wed, Jan 12, 2011 at 11:17 AM, Stack <[email protected]> > wrote: > > >> >>>> > > >> >>>>> On Tue, Jan 11, 2011 at 2:34 PM, Wayne <[email protected]> > wrote: > > >> >>>>>> We have very frequent cluster wide pauses that stop all > > >> >>>>>> reads and > > >> >>>>> writes > > >> >>>>>> for seconds. > > >> >>>>> > > >> >>>>> All reads and all writes? > > >> >>>>> > > >> >>>>> I've seen the pause too for writes. Its something I've > > >> >>>>> always > > meant > > >> >>>>> to look into. Friso postulates one cause. Another that > > >> >>>>> we've > > >> talked > > >> >>>>> of is a region taking a while to come back on line after a > > >> >>>>> split > > or > > >> a > > >> >>>>> rebalance for whatever reason. Client loading might be 'random' > > >> >>>>> spraying over lots of random regions but they all get stuck > > waiting > > >> on > > >> >>>>> one particular region to come back online. > > >> >>>>> > > >> >>>>> I suppose reads could be blocked for same reason if all are > > >> >>>>> trying > > >> to > > >> >>>>> read from the offlined region. > > >> >>>>> > > >> >>>>> What version of hbase are you using? Splits should be faster > > >> >>>>> in > > >> 0.90 > > >> >>>>> now that the split daughters come up on the same region. > > >> >>>>> > > >> >>>>> Sorry I don't have a better answer for you. Need to dig in. > > >> >>>>> > > >> >>>>> File a JIRA. If you want to help out some, stick some data > > >> >>>>> up in > > >> it. > > >> >>>>> Some suggestions would be to enable logging of when we > lookup > > region > > >> >>>>> locations in client and then note when requests go to zero. > > >> >>>>> Can > > you > > >> >>>>> figure what region the clients are waiting on (if they are > > >> >>>>> waiting > > >> on > > >> >>>>> any). If you can pull out a particular one, try and elicit > > >> >>>>> its history at time of blockage. Is it being moved or > > >> >>>>> mid-split? I suppose it makes sense that bigger regions > > >> >>>>> would make the > > situation > > >> >>>>> 'worse'. I can take a look at it too. > > >> >>>>> > > >> >>>>> St.Ack > > >> >>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> > > >> >>>>> We are constantly loading data to this cluster of 10 nodes. > > >> >>>>>> These pauses can happen as frequently as every minute but > > sometimes > > >> >> are > > >> >>>>> not > > >> >>>>>> seen for 15+ minutes. Basically watching the Region server > > >> >>>>>> list > > >> with > > >> >>>>> request > > >> >>>>>> counts is the only evidence of what is going on. All reads > > >> >>>>>> and > > >> writes > > >> >>>>>> totally stop and if there is ever any activity it is on the > > >> >>>>>> node > > >> >> hosting > > >> >>>>> the > > >> >>>>>> .META. table with a request count of region count + 1. This > > problem > > >> >>>>> seems to > > >> >>>>>> be worse with a larger region size. We tried a 1GB region > > >> >>>>>> size > > and > > >> >> saw > > >> >>>>> this > > >> >>>>>> more than we saw actual activity (and stopped using a larger > > region > > >> >> size > > >> >>>>>> because of it). We went back to the default region size and > > >> >>>>>> it > > was > > >> >>>>> better, > > >> >>>>>> but we had too many regions so now we are up to 512M for a > > >> >>>>>> region > > >> >> size > > >> >>>>> and > > >> >>>>>> we are seeing it more again. > > >> >>>>>> > > >> >>>>>> Does anyone know what this is? We have dug into all of the > > >> >>>>>> logs > > to > > >> >> find > > >> >>>>> some > > >> >>>>>> sort of pause but are not able to find anything. Is this an > > >> >>>>>> wal > > >> hlog > > >> >>>>> roll? > > >> >>>>>> Is this a region split or compaction? Of course our biggest > > >> >>>>>> fear > > is > > >> a > > >> >> GC > > >> >>>>>> pause on the master but we do not have java logging turned > > >> >>>>>> on > > with > > >> >> the > > >> >>>>>> master to tell. What could possibly stop the entire cluster > > >> >>>>>> from > > >> >> working > > >> >>>>> for > > >> >>>>>> seconds at a time very frequently? > > >> >>>>>> > > >> >>>>>> Thanks in advance for any ideas of what could be causing this. > > >> >>>>>> > > >> >>>>> > > >> >>>> > > >> >>>> > > >> >>> > > >> >> > > >> > > >> > > > > >
