[
https://issues.apache.org/jira/browse/HBASE-28257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-28257:
----------------------------------
Description:
*Steps to reproduce:*
# Make an edit to a region.
# Wait 1 hour + 10 seconds (default value of
{{hbase.regionserver.optionalcacheflushinterval}} plus
{{hbase.regionserver.flush.check.period}}.)
# Make a very large number of edits to the region (i.e. >= 1GB, pressure the
memstore.)
*Expected:*
Memstore pressure leads to flushes.
*Result:*
The PeriodicMemstoreFlusher has scheduled a refresh with a random delay of 0-5
minutes (default for
{{hbase.regionserver.periodicmemstoreflusher.rangeofdelayseconds}}.) Memstore
pressure flushes are blocked by the scheduled delayed flush. Client receives
0-5 minutes of {{RegionTooBusyExceptions}} until the delayed flush executes.
*Logs:*
2023-12-13 06:00:13,573 INFO regionserver.HRegionServer - MemstoreFlusherChore
requesting flush of <REGION> because <CF> has an old edit so flush to free WALs
after random delay 166761 ms
2023-12-13 06:00:53,219 DEBUG regionserver.HRegion - Flush requested on <REGION>
2023-12-13 06:01:47,694 WARN regionserver.HRegion - Region is too busy due to
exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException:
Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
...
repeats
...
2023-12-13 06:01:52,223 WARN regionserver.HRegion - Region is too busy due to
exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException:
Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
...
2023-12-13 06:03:00,340 INFO regionserver.HRegion - Flushing <REGION> 1/1
column families, dataSize=534.77 MB heapSize=1.00 GB
2023-12-13 06:03:09,634 INFO regionserver.HRegion - Finished flush of dataSize
~534.77 MB/560744948, heapSize ~1.00 GB/1073816296, currentSize=0 B/0 for
<REGION> in 9294ms, sequenceid=21310753, compaction requested=false
Note also this is the same cause as discussed in HBASE-16030 conversation
https://issues.apache.org/jira/browse/HBASE-16030?focusedCommentId=15340153&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15340153
was:
*Steps to reproduce:*
# Make an edit to a region.
# Wait 1 hour (default value of hbase.regionserver.optionalcacheflushinterval.)
# Make a very large number of edits to the region (i.e. >= 1GB, pressure the
memstore.)
*Expected:*
Memstore pressure leads to flushes.
*Result:*
The PeriodicMemstoreFlusher has scheduled a refresh with a random delay of 0-5
minutes (default for
hbase.regionserver.periodicmemstoreflusher.rangeofdelayseconds.) Memstore
pressure flushes are blocked by the scheduled delayed flush. Client receives
0-5 minutes of RegionTooBusyExceptions until the delayed flush executes.
*Logs:*
2023-12-13 06:00:13,573 INFO regionserver.HRegionServer - MemstoreFlusherChore
requesting flush of <REGION> because <CF> has an old edit so flush to free WALs
after random delay 166761 ms
2023-12-13 06:00:53,219 DEBUG regionserver.HRegion - Flush requested on <REGION>
2023-12-13 06:01:47,694 WARN regionserver.HRegion - Region is too busy due to
exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException:
Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
...
repeats
...
2023-12-13 06:01:52,223 WARN regionserver.HRegion - Region is too busy due to
exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException:
Over memstore limit=1.0 G, regionName=<REGION>, server=<SERVER>
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
...
2023-12-13 06:03:00,340 INFO regionserver.HRegion - Flushing <REGION> 1/1
column families, dataSize=534.77 MB heapSize=1.00 GB
2023-12-13 06:03:09,634 INFO regionserver.HRegion - Finished flush of dataSize
~534.77 MB/560744948, heapSize ~1.00 GB/1073816296, currentSize=0 B/0 for
<REGION> in 9294ms, sequenceid=21310753, compaction requested=false
Note also this is the same cause as discussed in HBASE-16030 conversation
https://issues.apache.org/jira/browse/HBASE-16030?focusedCommentId=15340153&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15340153
> Memstore flushRequest can be blocked by a delayed flush scheduled by
> PeriodicMemstoreFlusher, RegionTooBusyExceptions for up to 5 minutes
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-28257
> URL: https://issues.apache.org/jira/browse/HBASE-28257
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 2.0.0, 3.0.0
> Reporter: David Manning
> Assignee: David Manning
> Priority: Minor
>
> *Steps to reproduce:*
> # Make an edit to a region.
> # Wait 1 hour + 10 seconds (default value of
> {{hbase.regionserver.optionalcacheflushinterval}} plus
> {{hbase.regionserver.flush.check.period}}.)
> # Make a very large number of edits to the region (i.e. >= 1GB, pressure the
> memstore.)
> *Expected:*
> Memstore pressure leads to flushes.
> *Result:*
> The PeriodicMemstoreFlusher has scheduled a refresh with a random delay of
> 0-5 minutes (default for
> {{hbase.regionserver.periodicmemstoreflusher.rangeofdelayseconds}}.) Memstore
> pressure flushes are blocked by the scheduled delayed flush. Client receives
> 0-5 minutes of {{RegionTooBusyExceptions}} until the delayed flush executes.
> *Logs:*
> 2023-12-13 06:00:13,573 INFO regionserver.HRegionServer -
> MemstoreFlusherChore requesting flush of <REGION> because <CF> has an old
> edit so flush to free WALs after random delay 166761 ms
> 2023-12-13 06:00:53,219 DEBUG regionserver.HRegion - Flush requested on
> <REGION>
> 2023-12-13 06:01:47,694 WARN regionserver.HRegion - Region is too busy due to
> exceeding memstore size limit.
> org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G,
> regionName=<REGION>, server=<SERVER>
> at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
> ...
> repeats
> ...
> 2023-12-13 06:01:52,223 WARN regionserver.HRegion - Region is too busy due to
> exceeding memstore size limit.
> org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=1.0 G,
> regionName=<REGION>, server=<SERVER>
> at org.apache.hadoop.hbase.regionserver.HRegion.checkResources
> ...
> 2023-12-13 06:03:00,340 INFO regionserver.HRegion - Flushing <REGION> 1/1
> column families, dataSize=534.77 MB heapSize=1.00 GB
> 2023-12-13 06:03:09,634 INFO regionserver.HRegion - Finished flush of
> dataSize ~534.77 MB/560744948, heapSize ~1.00 GB/1073816296, currentSize=0
> B/0 for <REGION> in 9294ms, sequenceid=21310753, compaction requested=false
> Note also this is the same cause as discussed in HBASE-16030 conversation
> https://issues.apache.org/jira/browse/HBASE-16030?focusedCommentId=15340153&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15340153
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
