[
https://issues.apache.org/jira/browse/KUDU-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17171665#comment-17171665
]
Andrew Wong commented on KUDU-3180:
-----------------------------------
It makes sense to prefer larger mem-stores over smaller mem-stores. However, I
also think it's worth keeping KUDU-3002 in mind – if we start using that as a
policy, the likelihood that we starve the flushing of smaller mem-stores goes
up within the same tablet, e.g. we may anchor some WAL segments forever because
we never flush our DMSs, and WAL space usage can thus remain high forever. It's
worth considering an approach that examines the bytes in the WAL retained
instead of size so we don't fall into that trap.
Another thing that confuses me is the fact that in your screenshot, the "Logs
retained" are showing 0B for some mem-stores. It doesn't make sense to have a
mem-store around that anchors 0 WALs, since the whole point of the WALs is to
rebuild in-memory state. What version of Kudu are you running?
> kudu don't always prefer to flush MRS/DMS that anchor more memory
> -----------------------------------------------------------------
>
> Key: KUDU-3180
> URL: https://issues.apache.org/jira/browse/KUDU-3180
> Project: Kudu
> Issue Type: Bug
> Reporter: YifanZhang
> Priority: Major
> Attachments: image-2020-08-04-20-26-53-749.png,
> image-2020-08-04-20-28-00-665.png
>
>
> Current time-based flush policy always give a flush op a high score if we
> haven't flushed for the tablet in a long time, that may lead to starvation of
> ops that could free more memory.
> We set -flush_threshold_mb=32, -flush_threshold_secs=1800 in a cluster, and
> find that some small MRS/DMS flushes has a higher perf score than big MRS/DMS
> flushes and compactions, which seems not so reasonable.
> !image-2020-08-04-20-26-53-749.png|width=1424,height=317!!image-2020-08-04-20-28-00-665.png|width=1414,height=327!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
