zealchen opened a new issue, #1608: URL: https://github.com/apache/horaedb/issues/1608
### Describe This Problem The current manifest snapshot archiving process requires deduplication during every merge operation involving a single snapshot file and multiple SST files. Let's first look at the snapshot archiving procedure: <img width="1011" alt="image" src="https://github.com/user-attachments/assets/96595179-b4d3-4222-8ced-c72d9704d33b" /> The duplication scenario can only exist when the manifest merger reads an old metafile that has already gone through the merging procedure once. This consistent problem is due to the failure to delete the meta files after the persistence of the snapshot file. The failure itself can lie in many reasons like network problem or crashing of the running process. This issue is an idempotency problem, or in other words, the merge, write, and delete operations must be performed within a single transaction. ### Proposal We could design a batch operation transaction mechanism to ensure that: 1. If the last merge procedure was successfully completed, there is no need to perform deduplication. 2. If the last merge procedure was partially completed, deduplication is required. 3. If it is unclear whether the last merge procedure was successful or not, deduplication should be performed. To determine which scenario applies, we need to record the states of the merge operations and store them locally. This allows us to handle the scenarios as follows: a. If the state file does not exist, proceed to scenario 3. b. If the state file indicates that the delete operation is complete, proceed to scenario 1. c. If the state file indicates that the snapshot was successfully stored but the delete operation is incomplete, proceed to scenario 2. The pseudo code is like below: ```rust fn do_merge() { let sstfiles = ...; let snapshot = ...; let ops = MergeOps::new(snapshot, sstfiles); { let ops_guard = BatchOpsGuard::transaction(ops); ops_guard.execute(|| batch_ops.merge_sst_meta()); ops_guard.execute(|| batch_ops.store_snapshot()); ops_guard.execute(|| batch_ops.delete_sst_meta()); } } trait BatchOps { type State; fn need_redo(state: State) -> bool; fn redo(state: State) -> Result<()>; fn start() -> State; fn finish() -> State; } impl BatchOps for MergeOps { type State = MergeState; fn need_redo(state: State) -> bool; fn redo(state: State) -> Result<()>; fn start() -> State; fn finish() -> State; fn merge_sst_meta(..) -> State; fn store_snapshot(..) -> State; fn delete_sst_meta(..) -> State; } impl BatchOpsGuard { fn transaction(ops) -> Self where ops: BatchOps; { let state = read_file(..); if ops.need_redo(state) { ops.redo(state); } let state = ops.start(); save_file(state); } fn execute(&self, fn: F) where F: Fn -> BatchOps::State { let state = fn(); save_file(state); } } impl Drop for BatchOpsGuard { fn drop(&mut self) { let state = self.ops.finish(); save_file(state); } } ``` ### Additional Context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@horaedb.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@horaedb.apache.org For additional commands, e-mail: commits-h...@horaedb.apache.org