The error disappeared after changing write buffer from 20MB to 2MB. Thanks for the help!
Jianshi On Wed, Mar 4, 2015 at 12:12 AM, Jean-Marc Spaggiari < [email protected]> wrote: > It depends on how you manage your connection, your table and your puts. If > it works for you with reducing the batch buffer size, then just keep it the > way it is... > > JM > > 2015-03-03 11:10 GMT-05:00 Jianshi Huang <[email protected]>: > > > Yes, looks like reducing the batch buffer size works (still validating). > > > > But why setAutoFlush(false) is harmful here? I just want maximum write > > speed. > > > > Jianshi > > > > On Tue, Mar 3, 2015 at 10:54 PM, Jean-Marc Spaggiari < > > [email protected]> wrote: > > > > > Let HBase manage the flushes for you. Remove > > edgeTable.setAutoFlush(false) > > > and maybe reduce your batch size. > > > > > > I don't think that increasing the memstore is the good way to go. Sound > > > more like a plaster on the issue than a good fix (for me). > > > > > > JM > > > > > > 2015-03-03 9:43 GMT-05:00 Ted Yu <[email protected]>: > > > > > > > Default value for hbase.regionserver.global.memstore.size is 0.4 > > > > > > > > Meaning Maximum size of all memstores in the region server before new > > > > updates > > > > are blocked and flushes are forced is 7352m which is lower than 774m. > > > > > > > > You can increase the value for > hbase.regionserver.global.memstore.size > > > > > > > > Please also see if you can distribute the writes to the underlying > > region > > > > so that the region's use of memstore comes down. > > > > > > > > Cheersx > > > > > > > > On Tue, Mar 3, 2015 at 12:07 AM, Jianshi Huang < > > [email protected]> > > > > wrote: > > > > > > > > > Hi Ted, > > > > > > > > > > Only one region server is problematic. > > > > > > > > > > hbase.regionserver.global.memstore.size is not set, the problematic > > > > region > > > > > is using 774m for memstore. > > > > > > > > > > Max heap is 18380m for all region servers. > > > > > > > > > > Jianshi > > > > > > > > > > > > > > > On Mon, Mar 2, 2015 at 10:59 PM, Ted Yu <[email protected]> > wrote: > > > > > > > > > > > What's the value for hbase.regionserver.global.memstore.size ? > > > > > > > > > > > > Did RegionTooBusyException happen to many regions or only a few > > > > regions ? > > > > > > > > > > > > How much heap did you give region servers ? > > > > > > > > > > > > bq. HBase version is 0.98.0.2.1.2.0-402 > > > > > > > > > > > > Yeah, this is a bit old. Please consider upgrading. > > > > > > > > > > > > Cheers > > > > > > > > > > > > On Mon, Mar 2, 2015 at 1:42 AM, Jianshi Huang < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm constantly facing "RegionTooBusyException: Above memstore > > > limit" > > > > > > errors > > > > > > > in one region server when writing data to HBase. > > > > > > > > > > > > > > I checked the region server log, and I've seen a lot of > warnings > > > > during > > > > > > the > > > > > > > data writes: > > > > > > > > > > > > > > WARN wal.fshlog couldn't find oldest seqNum for the region > > we're > > > > > about > > > > > > to > > > > > > > flush, ... > > > > > > > > > > > > > > Then HBase seem to flush the data and added it as a HStore > file. > > > > > > > > > > > > > > I also get a few warnings in client.ShortCircuitCache, says > > "could > > > > not > > > > > > load > > > > > > > ... due to InvalidToken exceptions. > > > > > > > > > > > > > > Anyone can give me hint what went wrong? > > > > > > > > > > > > > > My HBase version is 0.98.0.2.1.2.0-402, I'm using HDP 2.1, but > > the > > > > > > release > > > > > > > is a little bit old. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > -- > > > > > > > Jianshi Huang > > > > > > > > > > > > > > LinkedIn: jianshi > > > > > > > Twitter: @jshuang > > > > > > > Github & Blog: http://huangjs.github.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Jianshi Huang > > > > > > > > > > LinkedIn: jianshi > > > > > Twitter: @jshuang > > > > > Github & Blog: http://huangjs.github.com/ > > > > > > > > > > > > > > > > > > > > -- > > Jianshi Huang > > > > LinkedIn: jianshi > > Twitter: @jshuang > > Github & Blog: http://huangjs.github.com/ > > > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
