[
https://issues.apache.org/jira/browse/KUDU-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wong updated KUDU-3002:
------------------------------
Summary: DMS may never flush when under memory pressure and there are
on-going inserts (was: consider compactions as a mechanism to flush many DMSs)
> DMS may never flush when under memory pressure and there are on-going inserts
> -----------------------------------------------------------------------------
>
> Key: KUDU-3002
> URL: https://issues.apache.org/jira/browse/KUDU-3002
> Project: Kudu
> Issue Type: Improvement
> Components: perf, tablet
> Reporter: Andrew Wong
> Priority: Major
>
> When under memory pressure, we'll aggressively perform the maintenance
> operation that frees the most memory. Right now, the only ops that register
> memory are MRS and DMS flushes.
> In practice, this means a couple things:
> * In most cases, we'll prioritize flushing MRSs way ahead of flushing DMS,
> since updates are spread across many DMSs and will therefore tend to be
> small, whereas any non-trivial insert workload will well up into a single MRS
> for an entire tablet
> * We'll only flush a single DMS at a time to free memory. Because of this,
> and because we'll likely prioritize MRS flushes over DMS flushes, we may end
> up with a ton of tiny DMSs in a tablet that we'll never flush. This can end
> up bloating the WALs because each DMS may be anchoring some WAL segments.
> A couple thoughts on small things we can do to improve this:
> * Register the DMS size as ram anchored by a compaction. This will meant
> that we can schedule compactions to flush DMSs en masse. This would still
> mean that we could end up always prioritizing MRS flushes, depending on how
> quickly we're inserting.
> * We currently register the amount disk space an LogGC would free up. We
> could do something similar, but register how many log anchors an op could
> release. This would be a bit trickier, since the log anchors aren't solely
> determined by the mem-stores (e.g. we'll anchor segments to catch up slow
> followers).
> * Introduce a new op (or change the flush DMS op) that would flush as many
> DMSs as we can for a given tablet.
> Between these, the first seems like it'd be an easy win.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
