Alexey Serbin has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19281
Change subject: WIP KUDU-3406 memory budgeting for CompactRowSetsOp
......................................................................
WIP KUDU-3406 memory budgeting for CompactRowSetsOp
This patch implements memory budgeting for performing rowset merge
compactions (i.e. CompactRowSetsOp maintenance operations).
The idea is to check whether it's enough memory left before reaching the
hard memory limit if starting a CompactRowSetsOp. An estimate for the
amount of memory necessary to perform the operation is based on the
total on-disk size of all UNDO deltas in rowsets selected for the
merge compaction. If there is enough memory, then the operation is
scheduled, otherwise the operation is postponed. Meanwhile, UNDO deltas
eventually become ancient and UndoDeltaBlockGCOp drop those, so the
amount of memory required to perform the operation decreases in the
long run.
Prior to this patch, the root cause of running out of memory when
performing CompactRowSetsOp was trying to allocate to much memory
at least due to the following factors:
* too many UNDO deltas might accumulate in rowsets selected for the
compaction operation because of the relatively high setting for the
--tablet_history_max_age_sec flag (7 days) and a particular workload
that issues many updates for rows in the same rowset
* even if it's a merge-like operation by its nature, the current
implementation of CompactRowSetsOp allocates all the memory
necessary to load the UNDO deltas at once, and it keeps all the
preliminary results in the memory as well before persisting
the result data to disk
* the current implementation of CompactRowSetsOp loads all the UNDO
deltas from the rowsets selected for compaction regardless whether
they are ancient or not; it discards of the data sourced from the
ancient deltas in the very end before persisting the result data
Ideally, CompactRowSetsOp should be refactored to merge the deltas
in participating rowsets sequentially, chunk by chunk, persisting the
intermediate results and allocating memory just for small bunch of
deltas, not loading all the deltas at once. A future patch should take
care of that, while this patch provides an interim approach using memory
budgeting on top of the current CompactRowSetsOp implementation as-is.
WIP:
* collect initial feedback
* perform more testing against real workloads
* integrate with the new test introduced in
https://gerrit.cloudera.org/#/c/19278/
Change-Id: I89c171284944831e95c45a993d85fbefe89048cf
---
M src/kudu/tablet/delta_tracker.cc
M src/kudu/tablet/delta_tracker.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_mm_ops.cc
M src/kudu/tablet/tablet_mm_ops.h
5 files changed, 185 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/81/19281/1
--
To view, visit http://gerrit.cloudera.org:8080/19281
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I89c171284944831e95c45a993d85fbefe89048cf
Gerrit-Change-Number: 19281
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <[email protected]>
