Mike Percy has uploaded a new change for review. http://gerrit.cloudera.org:8080/6063
Change subject: KUDU-1601. Delete ancient UNDO delta blocks in the background ...................................................................... KUDU-1601. Delete ancient UNDO delta blocks in the background This patch adds a maintenance manager background task that deletes "ancient" UNDO delta blocks, which means blocks that correspond to data that is considered no longer reachable and a candidate for garbage collection. The task only deletes entire blocks and so does not provoke write amplification. The task was written to operate at the server level because the UpdateStats() part of the task spends a budgeted amount of time opening (initializing) undo delta blocks in order to read the stats from their headers. Without reading the delta stats, it is impossible to know whether a given block is a candidate for GC. This patch includes the following: * Adds UNDO delta block GC task at TS / Master level * Adds a bunch of metrics, both at the tablet and TS level * Adds flags to enable / disable the GC task as well as flags to throttle it * Makes several minor improvements in the maintenance manager * Shut down MM ops before shutting down Tablet * Fix a few lint complaints * Only flush tablet metadata after doing all writes. TODO: This patch could use more tests, including the following: * Rowset-level unit test * Incorporate into mt-tablet-test * Incorporate into fuzz-test * Integration test utilizing the actual MM tasks (tserver, master) Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68 Woooo Change-Id: I8432548d4e6b6d5197c15737b9fae9f850f4de32 --- M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master.cc M src/kudu/master/master.h M src/kudu/server/server_base.h M src/kudu/tablet/delta_tracker.cc M src/kudu/tablet/delta_tracker.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/memrowset.h M src/kudu/tablet/mock-rowsets.h M src/kudu/tablet/rowset.h M src/kudu/tablet/rowset_metadata.cc M src/kudu/tablet/rowset_metadata.h M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet.h M src/kudu/tablet/tablet_history_gc-test.cc M src/kudu/tablet/tablet_metrics.cc M src/kudu/tablet/tablet_metrics.h M src/kudu/tserver/CMakeLists.txt M src/kudu/tserver/tablet_peer_lookup.h M src/kudu/tserver/tablet_server.cc M src/kudu/tserver/tablet_server.h M src/kudu/tserver/ts_tablet_manager.cc M src/kudu/tserver/ts_tablet_manager.h A src/kudu/tserver/tserver_mm_ops.cc A src/kudu/tserver/tserver_mm_ops.h M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h 30 files changed, 1,099 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/6063/1 -- To view, visit http://gerrit.cloudera.org:8080/6063 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8432548d4e6b6d5197c15737b9fae9f850f4de32 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Mike Percy <[email protected]>
