Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19413 )
Change subject: KUDU-3406 corrected estimate for ancient UNDO delta size ...................................................................... KUDU-3406 corrected estimate for ancient UNDO delta size When looking into micro-benchmark results produced by the $KUDU_HOME/src/kudu/scripts/benchmarks.sh script, I noticed that dense_node-itest showed 9-fold increase in the number of blocks under management. Even if the test disables GC of the ancient UNDO deltas (i.e. it runs with --enable_undo_delta_block_gc=false), that's not the expected behavior. It turned out the issue was in the way how DeltaTracker::EstimateBytesInPotentiallyAncientUndoDeltas() operated: it always treated a delta to be ancient if no stats were present. So, if a delta file was lazily loaded without stats being read, DeltaTracker assumed all its deltas were ancient. With the new behavior introduced in 1556a353e, it led to rowset merge compactions skipping the newly generated UNDO deltas since the estimate reported 100% of those deltas being ancient. While this was not detected by prior testing using various real-world scenarios involving some tangible amount of data written, tracking the history of stats emitted by dense_node-itest allowed to spot the issue. This patch addresses the issue, introducing different estimate modes for the method mentioned above and using proper modes in various contexts. I verified that with this patch added, the benchmark based on dense_node-itest now reports the number of blocks as it has been reporting for the longest span of its history. So I'm not adding any new tests with this patch. This is a follow-up to 1556a353e60c5d555996347cbd46d5e5a6661266. Change-Id: I17bddae86f84792caf14fb1e11a6e1c0d7a92b56 Reviewed-on: http://gerrit.cloudera.org:8080/19413 Tested-by: Kudu Jenkins Reviewed-by: Attila Bukor <[email protected]> --- M src/kudu/tablet/delta_tracker.cc M src/kudu/tablet/delta_tracker.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.cc 8 files changed, 73 insertions(+), 29 deletions(-) Approvals: Kudu Jenkins: Verified Attila Bukor: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/19413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17bddae86f84792caf14fb1e11a6e1c0d7a92b56 Gerrit-Change-Number: 19413 Gerrit-PatchSet: 5 Gerrit-Owner: Alexey Serbin <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Attila Bukor <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yifan Zhang <[email protected]>
