Todd Lipcon has submitted this change and it was merged. Change subject: generic_iterators: avoid holding completed iters in merge/union ......................................................................
generic_iterators: avoid holding completed iters in merge/union Previously, the MergeIterator and UnionIterator would hold on to all of the iterators they consume until the entire process completed. Each such iterator may hold on to blocks in the block cache (eg dictionary blocks) and consume a fair amount of memory. Without this patch, I noticed that issuing a scan caused the block cache consumed memory to slowly increase until the scan finished, at which point it would quickly drop. With the patch, I no longer see this behavior (memory usage is constant). The one feature lost here is the stringification of MergeIterator which previously included all of the sub-iterators. We only used this in one place in a VLOG(1) message anyway, and for typical merges across thousands of DRS iterators, this wasn't likely to be particularly useful anyway. Along the way, I did a bit of cleanup in cfile_set.cc to use unique_ptr instead of pre-C++11 raw pointers. Change-Id: I8e068c256c5855efeff5c6325dbb5fbf770eb900 Reviewed-on: http://gerrit.cloudera.org:8080/6460 Tested-by: Kudu Jenkins Reviewed-by: David Ribeiro Alves <[email protected]> --- M src/kudu/common/generic_iterators.cc M src/kudu/common/generic_iterators.h M src/kudu/integration-tests/update_scan_delta_compact-test.cc M src/kudu/tablet/cfile_set.cc M src/kudu/tablet/cfile_set.h M src/kudu/tablet/rowset.cc M src/kudu/tablet/tablet.cc 7 files changed, 106 insertions(+), 82 deletions(-) Approvals: David Ribeiro Alves: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/6460 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8e068c256c5855efeff5c6325dbb5fbf770eb900 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: David Ribeiro Alves <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <[email protected]>
