99% of the time it's because there's a lot of regions in the compaction queue and the one that needs compaction is far behind. See this related jira https://issues.apache.org/jira/browse/HBASE-2646
One way around that problem is setting hbase.hstore.blockingStoreFiles higher than 7, here we have it a 16 but our machines are quite beefy. J-D On Fri, Sep 24, 2010 at 9:11 AM, Dan Harvey <[email protected]> wrote: > Hey, > > We seem to have come across a bug in hbase with how it is flushing the > memstore when it is full, I think it's related to this > https://issues.apache.org/jira/browse/HBASE-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672717#action_12672717but > I'm not sure. > > We are currently writing updates to a lot of the rows in a table > and unfortunately are writing to the keys in order so a region gets a lot of > writes at a short time, I know this isn't good practice but we didn't > realise it wouldn't happen in this task! > > So after quite a few writes we get :- > > 2010-09-24 16:02:16,619 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 13 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:16,652 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 99 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:16,701 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 25 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,197 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 89 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,269 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 73 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 92 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,357 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 95 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 > 77: memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,544 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 18 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: > memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,574 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 56 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: > memstore size 130.2m is >= than blocking 128.0m size > 2010-09-24 16:02:17,722 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Blocking updates for 'IPC Server handler 74 on 60020' on region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: > memstore size 130.2m is >= than blocking 128.0m size > > Looking to the code the first put that gets blocked sets the memstore to be > flushed but nothing seems to happen until after 90 seconds which seems to be > a coded time out :- > > 2010-09-24 16:03:42,984 WARN > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Tried to hold up > flushing for compactions of region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 but > have waited longer than 90000ms, continuing > > Then the memstore for that region gets flushed in 2s and all the puts are > unblocked. > > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 74 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 56 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 92 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 18 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 25 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 89 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 73 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 95 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 13 on 60020' > 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Unblocking updates for region > canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC > Server handler 99 on 60020' > > Is there a reason for hbase blocking for this long with the flushing? or > does it seem to be a bug? > > If no one else is getting this is there maybe a way to reduce the chance of > this happening to a region? > > Thanks, > > -- > Dan Harvey | Datamining Engineer > www.mendeley.com/profiles/dan-harvey > > Mendeley Limited | London, UK | www.mendeley.com > Registered in England and Wales | Company Number 6419015 >
