Hi William, The disk layout of IoTDB looks similar to Ozone. Unfortunately, it is hard to have a general mechanism to handle such a complicated layout. Also, when the data size is large, we may want to have incremental snapshots instead of transferring everything.
Does IoTDB have a large snapshot size? Ozone disables InstallSnapshot (raft.server.log.appender.install.snapshot.enabled) and uses InstallSnapshotNotification. Then, the leader only notifies followers to install a snapshot. The followers use its own mechanism to install a snapshot. Currently, Ratis assumes all the directories (raft log, snapshot & tmp) are under the same root directory. We may support a more flexible directory layout (e.g. allowing directories under different roots) if it helps. Tsz-Wo On Thu, Oct 20, 2022 at 9:46 PM William Song <[email protected]> wrote: > Hi, > > IoTDB StateMachine manages both in-memory states and data files persisted > on disk. These data files are stored in another directory and even in > another disk. When asked to take a snapshot, instead of copying these data > files to the statemachine dir(which consumes lots of unnecessary disk space > and time), statemachine only records absolute paths linking to these files. > > When leader installs snapshot to follower, the statemachine will provide > the absolute paths of all data files in FileInfo. The leader seems to > assume that all these files are under statemachine dir, and calculates its > relative path to sm and uses its as filename [1]. This will produce messing > relative paths like ../../../root/disk2/data/a.tsfile. > > The receiving follower uses the relative filename to create files holding > the incoming snapshot chunks. As a result, the file may be names as > sm/tmp/snapshot-uuid/../../../root/dik2/data/a.tsfile, sitting outside the > tmp dir. Later the rename operation won’t place these files to the correct > place and the statemachine can’t find the snapshot after InstallSnapshot. > > What’s the proper solution to handle this? > > Regards, > William > > > [1] > https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/FileChunkReader.java#L55 > > [2] > https://github.com/apache/ratis/blob/75a1071a3c62a5a1a09c356cc1cdf281cc506baf/ratis-server/src/main/java/org/apache/ratis/server/storage/SnapshotManager.java#L90 > >
