Mike Percy has uploaded a new patch set (#2). Change subject: KUDU-1607. Unpin tablet flush after failed bootstrap ......................................................................
KUDU-1607. Unpin tablet flush after failed bootstrap We have heard reports that, in certain scenarios, a failed tablet is unable to be deleted. After some investigation, I determined that this is because we neglect to unpin the Tablet when tablet bootstrap fails. When a table is being deleted, we delete each tablet superblock by calling TabletMetadata::DeleteSuperBlock(). This method double-checks that no orphaned blocks remain referenced to ensure we don't leak disk space. That requires TabletMetadata::DeleteTabletData() to be called first, which invokes Flush() twice to ensure that no orphaned blocks are referenced on disk upon returning. However, if the tablet is pinned, Flush() silently becomes a no-op (except for a log message that is printed) and so DeleteSuperBlock() also fails, resulting in the tablet not being fully deleted and the following warning message being written to the log file: W0317 13:05:19.324373 13242 ts_tablet_manager.cc:634] Invalid argument: Unable to delete on-disk data from tablet d1b857ddaa0743c79c9bcbd9b2fda174: The metadata for tablet d1b857ddaa0743c79c9bcbd9b2fda174 still references orphaned blocks. Call DeleteTabletData() first This patch makes the following changes: 1. Always unpin the tablet after pinning it (using RAII). 2. Add a new itest that fails without that change, reproducing the above warning message. 3. Adds a little more test infra to MiniCluster (and the MiniClusterBase interface) to make it easier to use the helper functions in cluster_itest_util.h in MiniCluster-based tests. Change-Id: Id274c6ee1da75bc6f92ab91c0a01edaa009b8962 --- M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/external_mini_cluster.cc M src/kudu/integration-tests/external_mini_cluster.h M src/kudu/integration-tests/mini_cluster.cc M src/kudu/integration-tests/mini_cluster.h M src/kudu/integration-tests/mini_cluster_base.h A src/kudu/integration-tests/tablet_delete-itest.cc M src/kudu/tablet/tablet_bootstrap.cc M src/kudu/tserver/mini_tablet_server.cc M src/kudu/tserver/mini_tablet_server.h 10 files changed, 188 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/6422/2 -- To view, visit http://gerrit.cloudera.org:8080/6422 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id274c6ee1da75bc6f92ab91c0a01edaa009b8962 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Mike Percy <[email protected]> Gerrit-Reviewer: David Ribeiro Alves <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
