[
https://issues.apache.org/jira/browse/KUDU-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920277#comment-16920277
]
Andrew Wong commented on KUDU-2929:
-----------------------------------
That's a good point, even registering the DeltaMemStore memory would be more
accurate than what we have today.
In cases where there are no updates, I wonder if we can estimate the final
memory used by the final DRS as the sum of the largest input CFileReaders
memory footprint per column, blooms, ad-index indexes, etc. and registering
that estimate towards the anchored memory when we update compaction stats.
My main concern is that if we _do_ schedule a compaction, we may end up going
above the hard memory limit, but looking at how lax we are with memory limits
today, maybe it's acceptable to use more memory on an op that should free more
memory.
> Don't starve compactions under memory pressure
> ----------------------------------------------
>
> Key: KUDU-2929
> URL: https://issues.apache.org/jira/browse/KUDU-2929
> Project: Kudu
> Issue Type: Improvement
> Components: perf, tablet
> Reporter: Andrew Wong
> Priority: Major
>
> When a server is under memory pressure, the maintenance manager exclusively
> will look for the maintenance op that frees up the most memory. Some
> operations, like compactions, do not register any amount of "anchored memory"
> and effectively don't qualify for consideration.
> This means that when a tablet server is under memory pressure, compactions
> will never be scheduled, even though compacting may actually end up reducing
> memory (e.g. combining many rowsets-worth of CFileReaders into a single
> rowset). While it makes sense to prefer flushes to compactions, it probably
> doesn't make sense to do nothing vs compact.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
