Hello Zoltan Martonka, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24330
to look at the new patch set (#2).
Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n)
......................................................................
KUDU-3429 Fix rowset compaction memory accounting (2/n)
The patch fixes the following:
- 'delta_blocks_mem_size()' does double accounting of the CFile block
buffer by adding Slice::val again when it is already accounted for
inside DeltaFileIterator::delta_blocks_mem_size_. The Slice::val field
is a zero-copy view of CFile block buffer and not a separate memory
area where block data is stored. Remove additional buffer sizes.
- Keep memory accounting calculation for prepare_deltas_ separate from
delta_blocks_mem_size_ to avoid scenarios where delta block buffers
are freed (inside FreeDeltaBlocks) while struct overhead for entries
in prepared_deltas_ remain until preparation of next batch starts.
So, even after clearing of delta blocks , memory_footprint() still
reflects surviving PreparedDelta struct overhead.
- 'peak_mem_usage' doesn't show the correct peak of memory usage of a
rowset compaction operation. It leaves out base-data arena and
ApplyMutations arena contributions that the tracker does capture.
MergeCompactionInput::max_memory_usage_ is sampled at FinishBlock()
time when all the DRS delta blocks are not residing in memory
simultaneously. Instead, make use of tracker's peak_consumption() that
takes into consideration all the big consumers of memory during a
rowset operation except base row data arena that is significantly low
in memory usage as compared to other consumers.
- After fixing the double-counting in delta_blocks_mem_size(), tracker
value is lower and the old 3 MB constant is no longer reliably
exceeded for tests in compaction-highmem-test. Set the threshold
(compaction_mem_usage_approx) to a value that is reliably below the
minimum tracked allocation during compaction run.
- Pre-account for the arena's initial component so that the first
PrepareBlock's Reset() produces a correctly-signed delta.
Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
---
M src/kudu/tablet/compaction-highmem-test.cc
M src/kudu/tablet/compaction.cc
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/deltafile.cc
M src/kudu/tablet/deltafile.h
M src/kudu/tablet/tablet.cc
6 files changed, 44 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24330/2
--
To view, visit http://gerrit.cloudera.org:8080/24330
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456
Gerrit-Change-Number: 24330
Gerrit-PatchSet: 2
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
