Andrew Wong has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14760 )
Change subject: WIP KUDU-2993: don't require update_dirs to fix directory inconsistencies ...................................................................... WIP KUDU-2993: don't require update_dirs to fix directory inconsistencies This patch removes the ENFORCE_CONSISTENCY behavior when opening the DataDirManager. By default, the directory will be opened with the UPDATE_ON_DISK behavior. To support this: - We can now tolerate failures when updating the PIMFs. - When we start up, we'll look at all the PIMFs, check if any are missing and create those that are, and update any that have a different 'all_uuids' field than what actually exists. Since we now always rewrite the PIMFs to be consistent, the "integrity check" is now gone. This check was previously useful to ensure that the 'all_uuids' fields matched for every PIMF, which ensured that every data directory that was expected to exist actually existed. This was important for a couple reasons: - When a single missing data directory spelled failure for the entire node, starting up with even a single "inconsistent" directory would break all tablets on the tserver. - The file block manager requires that the UUID indexes used by the DataDirManager are static. These indexes are defined by the ordering of the UUIDs in the PIMFs, so we used the integrity check to ensure the ordering was consistent across PIMFs. Now that Kudu tablets can start up with missing directories, the first reason isn't particularly enticing. The second is trickier to work around. To work around it, I've kept the essence of the UUID picking, though I've made the "integrity checking" virtually non-existent. For the log block manager, I've made the UUID indexing much simpler: rather than relying on the integrity check, we'll now always assign a PIMF a UUID, even if we couldn't read one from disk. WIP: - Decide what to do about `kudu fs update_dirs` Tests: - Adds a test to make sure that this doesn't completely break the file block manager. Given we don't expect heavy usage of the FBM, I didn't both doing extensive testing when the PIMFs are tampered with. - Updated a few tests that previously enforced consistency among PIMFs Change-Id: Ic3027e7edb5c60e96ced6160fec1a380b38353a5 --- M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager_util-test.cc M src/kudu/fs/block_manager_util.cc M src/kudu/fs/block_manager_util.h M src/kudu/fs/data_dirs.cc M src/kudu/fs/data_dirs.h M src/kudu/fs/error_manager.h M src/kudu/fs/fs.proto M src/kudu/fs/fs_manager-test.cc M src/kudu/fs/fs_manager.cc M src/kudu/server/server_base.cc 11 files changed, 450 insertions(+), 623 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/14760/3 -- To view, visit http://gerrit.cloudera.org:8080/14760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic3027e7edb5c60e96ced6160fec1a380b38353a5 Gerrit-Change-Number: 14760 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Wong <[email protected]>
