Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24330 )
Change subject: KUDU-3429 Fix rowset compaction memory accounting (2/n) ...................................................................... KUDU-3429 Fix rowset compaction memory accounting (2/n) The patch fixes the following: - 'delta_blocks_mem_size()' does double accounting of the CFile block buffer by adding Slice::val again when it is already accounted for inside DeltaFileIterator::delta_blocks_mem_size_. The Slice::val field is a zero-copy view of CFile block buffer and not a separate memory area where block data is stored. Remove additional buffer sizes. - Keep memory accounting calculation for prepare_deltas_ separate from delta_blocks_mem_size_ to avoid scenarios where delta block buffers are freed (inside FreeDeltaBlocks) while struct overhead for entries in prepared_deltas_ remain until preparation of next batch starts. So, even after clearing of delta blocks , memory_footprint() still reflects surviving PreparedDelta struct overhead. - 'peak_mem_usage' doesn't show the correct peak of memory usage of a rowset compaction operation. It leaves out base-data arena and ApplyMutations arena contributions that the tracker does capture. MergeCompactionInput::max_memory_usage_ is sampled at FinishBlock() time when all the DRS delta blocks are not residing in memory simultaneously. Instead, make use of tracker's peak_consumption() that takes into consideration all the big consumers of memory during a rowset merge compaction operation. - After fixing the double-counting in delta_blocks_mem_size(), tracker value is lower and the old '3 MB' constant is no longer reliably exceeded for tests in compaction-highmem-test. Set the threshold (compaction_mem_usage_approx) to a value that is reliably below the minimum tracked allocation during compaction run. - Pre-account for the arena's initial component so that the first PrepareBlock's Reset() produces a correctly-signed delta. Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456 Reviewed-on: http://gerrit.cloudera.org:8080/24330 Tested-by: Marton Greber <[email protected]> Reviewed-by: Zoltan Martonka <[email protected]> Reviewed-by: Marton Greber <[email protected]> --- M src/kudu/tablet/compaction-highmem-test.cc M src/kudu/tablet/compaction.cc M src/kudu/tablet/delta_store.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/tablet.cc 6 files changed, 44 insertions(+), 23 deletions(-) Approvals: Marton Greber: Looks good to me, approved; Verified Zoltan Martonka: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/24330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456 Gerrit-Change-Number: 24330 Gerrit-PatchSet: 4 Gerrit-Owner: Ashwani Raina <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Attila Bukor <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Reviewer: Zoltan Martonka <[email protected]>
