Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/4363
to look at the new patch set (#12).
Change subject: KUDU-1601. Delete ancient UNDO delta blocks in the background
......................................................................
KUDU-1601. Delete ancient UNDO delta blocks in the background
This patch adds a maintenance manager background task that deletes
"ancient" UNDO delta blocks, which means blocks that correspond to data
that is considered no longer reachable and a candidate for garbage
collection. The task only deletes entire blocks and so does not provoke
write amplification.
The task was written to operate at the server level (as opposed to the
tablet level) because the UpdateStats() part of the task spends a
budgeted amount of time opening (initializing) undo delta blocks in
order to read the stats from their headers. Without reading the delta
stats, it is impossible to know whether a given block is a candidate for
GC.
This patch includes the following:
* New UNDO delta block GC MM task running at the TS / Master level
* New UNDO delta block GC metrics, both at the tablet and the server level
* Flags to enable / disable the GC task as well as flags to throttle it
* A few minor improvements in the maintenance manager
* Fixes for a few preexisting clang-tidy lint complaints
A couple of implementation details:
* We shut down external MM ops on a tablet before shutting down the
tablet itself. This is enforced with a condition variable and a new
mutex in tablet.cc
* When performing undo delta GC in Tablet::DeleteAncientUndoDeltas(), we
only flush the tablet metadata after making the metadata changes
across all rowsets. This should be safe because we are not actually
modifying any data, we are simply removing references to blocks that
are no longer reachable by new scanners.
Includes the following tests:
* RowSet-level unit test in diskrowset-test
* Tablet-level functional test in tablet_history_gc-test
* Tablet-level concurrency test in mt-tablet-test
* Integration test utilizing the tserver-level MM task in
tablet_history_gc-itest
* Incorporated into RandomizedTabletHistoryGcITest in
tablet_history_gc-itest
Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68
---
M src/kudu/integration-tests/alter_table-test.cc
M src/kudu/integration-tests/full_stack-insert-scan-test.cc
M src/kudu/integration-tests/fuzz-itest.cc
M src/kudu/integration-tests/tablet_history_gc-itest.cc
M src/kudu/integration-tests/ts_tablet_manager-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/server/server_base.h
M src/kudu/tablet/delta_tracker.cc
M src/kudu/tablet/delta_tracker.h
M src/kudu/tablet/diskrowset-test.cc
M src/kudu/tablet/diskrowset.cc
M src/kudu/tablet/diskrowset.h
M src/kudu/tablet/memrowset.h
M src/kudu/tablet/mock-rowsets.h
M src/kudu/tablet/mt-tablet-test.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/rowset_metadata.cc
M src/kudu/tablet/rowset_metadata.h
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_history_gc-test.cc
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tserver/CMakeLists.txt
M src/kudu/tserver/tablet_peer_lookup.h
M src/kudu/tserver/tablet_server.cc
M src/kudu/tserver/tablet_server.h
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/tserver/ts_tablet_manager.h
M src/kudu/tserver/tserver-path-handlers.cc
A src/kudu/tserver/tserver_mm_ops.cc
A src/kudu/tserver/tserver_mm_ops.h
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
M src/kudu/util/maintenance_manager.h
40 files changed, 1,505 insertions(+), 141 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/4363/12
--
To view, visit http://gerrit.cloudera.org:8080/4363
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0309bf7acfb6d018860c80f354012c3500da5c68
Gerrit-PatchSet: 12
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Mike Percy <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: David Ribeiro Alves <[email protected]>
Gerrit-Reviewer: Jean-Daniel Cryans <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <[email protected]>
Gerrit-Reviewer: Tidy Bot
