Hello Tidy Bot, Kudu Jenkins, Andrew Wong, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14654
to look at the new patch set (#15).
Change subject: KUDU-2975: Spread WAL across multiple directories
......................................................................
KUDU-2975: Spread WAL across multiple directories
Add a new gflag named "fs_wal_dirs" to support spreading WAL across
multiple directories.
Use a new class named "WalDirManager" to manage the WAL dirs. We record
the tablet's WAL directory UUID into the metadata. We determine the
tablet WAL location based on the dir UUID recorded in metadata at
reboot.
If switch 'fs_wal_dir' to 'fs_wal_dirs', first we need use "kudu fs
update_dirs" tool to update the WAL dir. We should make sure that the
new configuration includes the old ones, otherwise some tablets may be
failed after startup. Because the tablet's metadata in old version had
no WAL dir, we look for the tablet's WAL directory under all the new WAL
directories. If found, we write the talbet's WAL into WalDirManager. But
it's not persistent.
Some WAL directories are allowed to be missing at startup. Or some disks
that hold WAL are allowed to be failed at startup. If the tablets
located on failed WAL directories, they can be recovered by master.
This modification has the following functions not completed at present:
1. WAL disk failure not support.
2. some tools about WAL not modify.
3. Lack tests.
The following functions may need improvement
1. the method of selecting WAL directorie for tablet
2. delete tablet's WAL directory UUID form metadata
Change-Id: Ied496804421d91ff1fa63d49979fde971071506e
---
M src/kudu/consensus/consensus_peers-test.cc
M src/kudu/consensus/consensus_queue-test.cc
M src/kudu/consensus/log-test-base.h
M src/kudu/consensus/log.cc
M src/kudu/consensus/log_cache-test.cc
M src/kudu/consensus/log_reader.cc
M src/kudu/consensus/raft_consensus_quorum-test.cc
M src/kudu/fs/CMakeLists.txt
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/data_dirs.h
M src/kudu/fs/fs.proto
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_manager.h
M src/kudu/fs/fs_report.cc
M src/kudu/fs/fs_report.h
A src/kudu/fs/wal_dirs-test.cc
A src/kudu/fs/wal_dirs.cc
A src/kudu/fs/wal_dirs.h
M src/kudu/integration-tests/mini_cluster_fs_inspector.cc
M src/kudu/integration-tests/mini_cluster_fs_inspector.h
M src/kudu/integration-tests/multidir_cluster-itest.cc
M src/kudu/integration-tests/open-readonly-fs-itest.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/integration-tests/tablet_copy-itest.cc
M src/kudu/integration-tests/timestamp_advancement-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/master/mini_master-test.cc
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
M src/kudu/mini-cluster/internal_mini_cluster.cc
M src/kudu/mini-cluster/internal_mini_cluster.h
M src/kudu/mini-cluster/mini_cluster.h
M src/kudu/server/server_base.cc
M src/kudu/tablet/metadata.proto
M src/kudu/tablet/tablet_bootstrap-test.cc
M src/kudu/tablet/tablet_bootstrap.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/mini_tablet_server-test.cc
M src/kudu/tserver/mini_tablet_server.cc
M src/kudu/tserver/mini_tablet_server.h
M src/kudu/tserver/tablet_copy_client-test.cc
M src/kudu/tserver/tablet_copy_client.cc
M src/kudu/tserver/tablet_server-test-base.cc
M src/kudu/tserver/tablet_server-test-base.h
M src/kudu/tserver/ts_tablet_manager.cc
M src/kudu/util/env_util.cc
M src/kudu/util/env_util.h
M src/kudu/util/path_util.h
52 files changed, 2,785 insertions(+), 288 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/14654/15
--
To view, visit http://gerrit.cloudera.org:8080/14654
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ied496804421d91ff1fa63d49979fde971071506e
Gerrit-Change-Number: 14654
Gerrit-PatchSet: 15
Gerrit-Owner: YangSong <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: YangSong <[email protected]>
