Hello Ashwani Raina, Yingchun Lai, Yifan Zhang, Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19281
to look at the new patch set (#2).
Change subject: WIP KUDU-3406 memory budgeting for CompactRowSetsOp
......................................................................
WIP KUDU-3406 memory budgeting for CompactRowSetsOp
This patch implements memory budgeting for performing rowset merge
compactions (i.e. CompactRowSetsOp maintenance operations).
The idea is to check whether it's enough memory left before reaching the
hard memory limit if starting a CompactRowSetsOp. An estimate for the
amount of memory necessary to perform the operation is based on the
total on-disk size of all UNDO deltas in rowsets selected for the
merge compaction. If there is enough memory, then the operation is
scheduled, otherwise the operation is postponed. Meanwhile, UNDO deltas
eventually become ancient and UndoDeltaBlockGCOp drop those, so the
amount of memory required to perform the operation decreases in the
long run.
Prior to this patch, the root cause of running out of memory when
performing CompactRowSetsOp was trying to allocate too much memory
at least due to the following factors:
* many UNDO deltas might accumulate in rowsets selected for the
compaction operation because of the relatively high setting for the
--tablet_history_max_age_sec flag (7 days) and a particular workload
that issues many updates for rows in the same rowset
* even if it's a merge-like operation by its nature, the current
implementation of CompactRowSetsOp allocates all the memory
necessary to load the UNDO deltas at once, and it keeps all the
preliminary results in the memory as well before persisting
the result data to disk
* the current implementation of CompactRowSetsOp loads all the UNDO
deltas from the rowsets selected for compaction regardless whether
they are ancient or not; it discards of the data sourced from the
ancient deltas in the very end before persisting the result data
Ideally, the current implementation of CompactRowSetsOp should be
refactored to merge the deltas in participating rowsets sequentially,
chunk by chunk, persisting the results and allocating memory just for
small bunch of processed deltas, not loading all the deltas at once.
A future patch should take care of that, while this patch provides an
interim approach using memory budgeting on top of the current
CompactRowSetsOp implementation as-is.
The newly introduced behavior is gated by the following two flags:
* rowset_compaction_memory_estimate_enabled: whether to enable memory
budgeting for CompactRowSetsOp (default is 'true').
* rowset_compaction_ancient_delta_threshold_enabled: whether to
check against the ratio of ancient UNDO deltas across rowsets
selected for compaction (default is 'true').
In addition, the following two flags allow for tweaking the new
behavior gated by the corresponding flags above:
* rowset_compaction_delta_memory_factor: the multiplication factor for
the size of UNDO deltas to estimate how much memory CompactRowSetsOp
would consume if operating on those deltas (default is 8.0)
* rowset_compaction_ancient_delta_max_ratio: the threshold for the
ratio of the data size in ancient UNDO deltas to the total data size
of UNDO deltas in the rowsets selected for merge compaction
WIP:
* collect initial feedback
* perform more testing against real workloads
* integrate with the new test introduced in
https://gerrit.cloudera.org/#/c/19278/
Change-Id: I89c171284944831e95c45a993d85fbefe89048cf
---
M src/kudu/tablet/delta_tracker.cc
M src/kudu/tablet/delta_tracker.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet_mm_ops-test.cc
M src/kudu/tablet/tablet_mm_ops.cc
M src/kudu/tablet/tablet_mm_ops.h
6 files changed, 187 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/81/19281/2
--
To view, visit http://gerrit.cloudera.org:8080/19281
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89c171284944831e95c45a993d85fbefe89048cf
Gerrit-Change-Number: 19281
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yifan Zhang <[email protected]>
Gerrit-Reviewer: Yingchun Lai <[email protected]>
