Ashwani Raina has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24330
Change subject: [WIP] KUDU-3429 Fix rowset compaction memory accounting (2/n) ...................................................................... [WIP] KUDU-3429 Fix rowset compaction memory accounting (2/n) The patch fixes the following: - 'delta_blocks_mem_size()' does double accounting of the CFile block buffer by adding Slice::val again when it is already accounted for inside DeltaFileIterator::delta_blocks_mem_size_. The Slice::val field is a zero-copy view of CFile block buffer and not a separate memory area where block data is stored. Remove additional buffer sizes. - Keep memory accounting calculation for prepare_deltas_ separate from delta_blocks_mem_size_ to avoid scenarios where delta block buffers are freed (inside FreeDeltaBlocks) while struct overhead for entries in prepared_deltas_ remain until preparation of next batch starts. So, even after clearing of delta blocks , memory_footprint() still reflects surviving PreparedDelta struct overhead. - 'peak_mem_usage' doesn't show the correct peak of memory usage of a rowset compaction operation. It leaves out base-data arena and ApplyMutations arena contributions that the tracker does capture. MergeCompactionInput::max_memory_usage_ is sampled at FinishBlock() time when all the DRS delta blocks are not residing in memory simultaneously. Instead, make use of tracker's peak_consumption() that takes into consideration all the big consumers of memory during a rowset operation except base row data arena that is significantly low in memory usage as compared to other consumers. TODO: - Explore including base data area memory consumption, either in current or a separate patch. Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456 --- M src/kudu/tablet/delta_store.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/tablet.cc 4 files changed, 21 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24330/1 -- To view, visit http://gerrit.cloudera.org:8080/24330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icc2133d38ed766f6834392a2e335d9246b511456 Gerrit-Change-Number: 24330 Gerrit-PatchSet: 1 Gerrit-Owner: Ashwani Raina <[email protected]>
