Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8978 )
Change subject: KUDU-2202: support for removing data directories (take two) ...................................................................... KUDU-2202: support for removing data directories (take two) This commit extends "kudu fs update_dirs" to allow the removal of data directories. A data dir may be removed provided there are no tablets configured to use it. The --force flag may be used to override this safety check; tablets that depend on the removed data dir will fail to bootstrap the next time the server is started. I'll be the first to admit that this functionality isn't terribly useful at the moment because Kudu deployments are configured to stripe data across all data directories, so removing a data dir will fail all tablets, which is equivalent to reformatting the node. Nevertheless, it will become more useful as users customize the new --fs_target_data_dirs_per_tablet gflag. This is the second attempt at implementing this functionality. The first used a different interface and was vulnerable to a data loss issue documented in KUDU-2202. Andrew fixed that in commit 47b81c4, which reopens the door for safely removing data directories. I considered a stronger check that only prohibits data dir removal if there were data blocks on the to-be-removed data dir, but decided against it: - It's rare to find a tablet configured to use a data dir but without any actual blocks on it. In fact, that's only likely for empty tables, or tables with empty range partitions. - We'd still need to rewrite any unaffected tablet superblocks and remove the data dir from their data dir groups. There are several interesting modifications here: 1. The DataDirManager and FsManager's update_on_disk boolean option has been converted into a tri-state. The new IGNORE_INCONSISTENCY state is used by the CLI tool to create a "speculative" FsManager using the new data dir configuration. This FsManager is used to check whether existing tablets depend on the data dir to be removed without making destructive changes to on-disk data structures. 3. The DataDirManager now checks for integrity after making on-disk changes. While not strictly necessary, this seems like a good sanity check to avoid introducing bugs. 4. The "kudu fs update_dirs" action was updated for data dir removal. It now accepts the --force option, without which removal will only be allowed if no tablets are configured to use the removed data dir. Change-Id: Iacb650115bcdf055542ef2777641a3201fc8e30b Reviewed-on: http://gerrit.cloudera.org:8080/8978 Reviewed-by: Andrew Wong <[email protected]> Tested-by: Kudu Jenkins --- M src/kudu/fs/data_dirs.cc M src/kudu/fs/data_dirs.h M src/kudu/fs/fs_manager-test.cc M src/kudu/fs/fs_manager.cc M src/kudu/fs/fs_manager.h M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_fs.cc 7 files changed, 441 insertions(+), 92 deletions(-) Approvals: Andrew Wong: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8978 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iacb650115bcdf055542ef2777641a3201fc8e30b Gerrit-Change-Number: 8978 Gerrit-PatchSet: 11 Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]>
