Andrew Wong created KUDU-3256:
---------------------------------
Summary: Limit the memory usage per transaction per tablet
Key: KUDU-3256
URL: https://issues.apache.org/jira/browse/KUDU-3256
Project: Kudu
Issue Type: Improvement
Components: transactions
Reporter: Andrew Wong
Currently, the transactions implementation stores all new inserts in a new MRS
per transaction per tablet. As transactions get larger and larger, or as there
are more transactions entering the system, this will result in memory pressure
across tablet servers. We should explore ways to limit the memory usage per
transaction, by either enforcing a memory limit per transaction participant, or
by flushing transactional MRSs before committing, per regular maintenance op
cadence (e.g. based on memory pressure, MRS size, time since last flush, etc.).
While it'd be significantly more complex, I'm more partial to the latter
approach – the mechanics to flush an MRS already exist, so why not use them? It
should be noted though that we would then need to update how bootstrapping is
handled by persisting a 'last_flushed_mrs_id' per transaction, similar to
what's done today for non-transactional MRSs. Additionally, the existing code
to swap in new disk rowsets atomically would need some thought to ensure
swapping in transactional rowsets while racing with a commit does the right
thing (i.e. if we flush the transactional MRS while committing, the end result
is the new DRSs should end up in the main rowset tree).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
