Ashwani Raina has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24092
Change subject: [WIP] [fs] keep orphaned blocks to cleanup in next flush
......................................................................
[WIP] [fs] keep orphaned blocks to cleanup in next flush
During flush, if orphaned blocks are there and for some reason those
blocks' records are not deleted (maybe due to some underlying transient
I/O error, etc), those blocks are never re-tried for deletion and erased
from the list of orphaned_blocks_. This can cause un-reclaimed space on
the persistent storage that can get accumulated over time to a big size.
This un-reclaimed space can only be reclaimed with user intervention by
running 'kudu fs check --repair'.
This patch addresses the issue by not removing those orphaned blocks
from the orphaned_blocks_ list, for which delete failed due to some
transient error. This ensures that when next flush is scheduled, the
metadata still contains those orphaned blocks in the list and delete
can be retried on those blocks to reclaim the space.
Highlights of the change:
- Add warning logs at CommitDeletedBlocks callers when the commit fails.
- Add per-block level error logs for blocks for which deletion record
could not be committed.
- Add these metrics to hold the orphan block deletion outcome:
* orphaned_blocks_cleaned
* orphaned_block_cleanup_failures
* orphaned_block_cleanup_failures_bytes
- Add unit tests to test three scenarios with metrics verification:
* Usual path where orphaned blocks are deleted with no error
* Induced I/O error where orphaned blocks are not deleted
* Two stages: First induce I/O error, verify metrics shows cleanup
failure count increased. Second remove induced error, verify metrics
shows cleaned count increased.
TODO:
1. Investigate KUDU-2635 angle for assumption in tablet deletion code
to expect orphan list be empty. If possible, run orphaned block
cleanup as part of tablet deletion workflow. Remedy to cleanup those
orphan blocks (for which deletion failed earlier) and tablet deletion
is bound to happen.
2. If adding the failed orphan blocks back to the list is not an option,
repurpose this patch to add metrics and necessary logging for better
diagnosis of scenarios where space runs out in orphaned blocks.
3. As part of this patch or separate one, add a background op to reclaim
the space left from orphaned blocks (KUDU-829)
Change-Id: Id386d9fc8d0900839e229e66772f35299b3ef2e9
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/tablet/tablet.cc
M src/kudu/tablet/tablet.h
M src/kudu/tablet/tablet_metadata-test.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
M src/kudu/tablet/tablet_metrics.cc
M src/kudu/tablet/tablet_metrics.h
8 files changed, 397 insertions(+), 33 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/92/24092/1
--
To view, visit http://gerrit.cloudera.org:8080/24092
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id386d9fc8d0900839e229e66772f35299b3ef2e9
Gerrit-Change-Number: 24092
Gerrit-PatchSet: 1
Gerrit-Owner: Ashwani Raina <[email protected]>
