Hi Tsz-Wo, If Ozone disables InstallSnapshot, how should leader sync data when a fresh new follower starts up and joins the cluster while the leader does not have the logs before latest snapshot?
William > 2022年10月21日 11:53，Tsz Wo Sze <[email protected]> 写道： > > Hi William, > > The disk layout of IoTDB looks similar to Ozone. Unfortunately, it is hard > to have a general mechanism to handle such a complicated layout. Also, when > the data size is large, we may want to have incremental snapshots instead of > transferring everything. > > Does IoTDB have a large snapshot size? > > Ozone disables InstallSnapshot > (raft.server.log.appender.install.snapshot.enabled) and uses > InstallSnapshotNotification. Then, the leader only notifies followers to > install a snapshot. The followers use its own mechanism to install a > snapshot. > > Currently, Ratis assumes all the directories (raft log, snapshot & tmp) are > under the same root directory. We may support a more flexible directory > layout (e.g. allowing directories under different roots) if it helps. > > Tsz-Wo > > > On Thu, Oct 20, 2022 at 9:46 PM William Song <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > IoTDB StateMachine manages both in-memory states and data files persisted on > disk. These data files are stored in another directory and even in another > disk. When asked to take a snapshot, instead of copying these data files to > the statemachine dir(which consumes lots of unnecessary disk space and time), > statemachine only records absolute paths linking to these files. > > When leader installs snapshot to follower, the statemachine will provide the > absolute paths of all data files in FileInfo. The leader seems to assume that > all these files are under statemachine dir, and calculates its relative path > to sm and uses its as filename [1]. This will produce messing relative paths > like ../../../root/disk2/data/a.tsfile. > > The receiving follower uses the relative filename to create files holding the > incoming snapshot chunks. As a result, the file may be names as > sm/tmp/snapshot-uuid/../../../root/dik2/data/a.tsfile, sitting outside the > tmp dir. Later the rename operation won’t place these files to the correct > place and the statemachine can’t find the snapshot after InstallSnapshot. > > What’s the proper solution to handle this? > > Regards, > William > > > [1] > https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/FileChunkReader.java#L55 > > <https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/FileChunkReader.java#L55> > > [2] > https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/SnapshotManager.java#L90 > > <https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/SnapshotManager.java#L90> >
