[
https://issues.apache.org/jira/browse/HBASE-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858513#action_12858513
]
ryan rawson commented on HBASE-2439:
------------------------------------
[[ Old comment, sent by email on Tue, 13 Apr 2010 15:28:24 -0700 ]]
The primary reason to do blocking store files is that compactions can
oome a server, so we protect ourselves. With 0.20 that should no
longer be the case. I have compacted 100 files of 10gb, so i am in
favor of removing this feature.
On Tue, Apr 13, 2010 at 3:01 PM, Kannan Muthukkaruppan (JIRA)
> HBase can get stuck if updates to META are blocked
> --------------------------------------------------
>
> Key: HBASE-2439
> URL: https://issues.apache.org/jira/browse/HBASE-2439
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Kannan Muthukkaruppan
> Assignee: Kannan Muthukkaruppan
> Fix For: 0.20.4, 0.20.5, 0.21.0
>
> Attachments: 2439_0.20_dont_block_meta.txt
>
>
> (We noticed this on a import-style test in a small test cluster.)
> If compactions are running slow, and we are doing a lot of region splits,
> then, since META has a much smaller hard-coded memstore flush size (16KB), it
> quickly accumulates lots of store files. Once this exceeds
> "hbase.hstore.blockingStoreFiles", flushes to META become no-ops. This causes
> METAs memstore footprint to grow. Once this exceeds
> "hbase.hregion.memstore.block.multiplier * 16KB", we block further updates to
> META.
> In my test setup:
> hbase.hregion.memstore.block.multiplier = 4.
> and,
> hbase.hstore.blockingStoreFiles = 15.
> And we saw messages of the form:
> {code}
> 2010-04-09 18:37:39,539 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Blocking updates for 'IPC Server handler 23 on 60020' on region .META.,,1:
> memstore size 64.2k is >= than blocking 64.0k size
> {code}
> Now, if around the same time the CompactSplitThread does a compaction and
> determines it is going split the region. As part of finishing the split, it
> wants to update META about the daughter regions.
> It'll end up waiting for the META to become unblocked. The single
> CompactSplitThread is now held up, and no further compactions can proceed.
> META's compaction request is itself blocked because the compaction queue will
> never get cleared.
> This essentially creates a deadlock and the region server is able to not
> progress any further. Eventually, each region server's CompactSplitThread
> ends up in the same state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
