Hey, We seem to have come across a bug in hbase with how it is flushing the memstore when it is full, I think it's related to this https://issues.apache.org/jira/browse/HBASE-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672717#action_12672717but I'm not sure.
We are currently writing updates to a lot of the rows in a table and unfortunately are writing to the keys in order so a region gets a lot of writes at a short time, I know this isn't good practice but we didn't realise it wouldn't happen in this task! So after quite a few writes we get :- 2010-09-24 16:02:16,619 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 13 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:16,652 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 99 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:16,701 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 25 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,197 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 89 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,269 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 73 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 92 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,357 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 95 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,12849849064 77: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,544 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 18 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,574 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 56 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: memstore size 130.2m is >= than blocking 128.0m size 2010-09-24 16:02:17,722 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC Server handler 74 on 60020' on region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477: memstore size 130.2m is >= than blocking 128.0m size Looking to the code the first put that gets blocked sets the memstore to be flushed but nothing seems to happen until after 90 seconds which seems to be a coded time out :- 2010-09-24 16:03:42,984 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Tried to hold up flushing for compactions of region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 but have waited longer than 90000ms, continuing Then the memstore for that region gets flushed in 2s and all the puts are unblocked. 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 74 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 56 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 92 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 18 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 25 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 89 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 73 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 95 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 13 on 60020' 2010-09-24 16:03:44,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: Unblocking updates for region canonical_documents,aaebeb30-b624-11df-a52e-0024e8453de6,1284984906477 'IPC Server handler 99 on 60020' Is there a reason for hbase blocking for this long with the flushing? or does it seem to be a bug? If no one else is getting this is there maybe a way to reduce the chance of this happening to a region? Thanks, -- Dan Harvey | Datamining Engineer www.mendeley.com/profiles/dan-harvey Mendeley Limited | London, UK | www.mendeley.com Registered in England and Wales | Company Number 6419015
