HTHou commented on code in PR #19287:
URL: https://github.com/apache/iotdb/pull/19287#discussion_r4581975669


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/checkpoint/CheckpointManager.java:
##########
@@ -223,19 +224,11 @@ public void fsyncCheckpointFile() {
             e);
         
DataNodeExceptionMetrics.getInstance().recordSuspiciousDiskException(e);
         CommonDescriptor.getInstance().getConfig().handleUnrecoverableError(e);
+        return;

Review Comment:
   **[P1] Propagate active checkpoint fsync failures to the WAL batch.** This 
early return leaves `checkCheckpointPublication()` clear, so `SyncBufferTask` 
can force the WAL and report success while its `CREATE_MEMORY_TABLE` record is 
still unconfirmed. I injected one `IOException` from the active checkpoint's 
`force()` with the roll threshold reached. The insert listener returned 
`SUCCESS` and the old checkpoint stayed active. Keeping only its last 
successfully forced bytes caused recovery to skip the insert. The base code 
forced a successor snapshot and replayed the same write. Please propagate the 
active checkpoint's fsync failure to the WAL batch or confirm a durable 
checkpoint with the mapping before acknowledging it.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/recover/CheckpointRecoverUtils.java:
##########
@@ -42,38 +45,35 @@ public static CheckpointInfo recoverMemTableInfo(File 
logDirectory) {
     }
     // desc sort by version id
     CheckpointFileUtils.descSortByVersionId(checkpointFiles);
-    // find last valid .checkpoint file and load checkpoints from it
-    long maxMemTableId = 0;
-    List<Checkpoint> checkpoints = null;
+    // Read back to the latest complete snapshot. Legacy rolls can have 
readable incremental
+    // checkpoints even if writing their snapshot failed.
+    Deque<CheckpointReader> readers = new ArrayDeque<>();
     for (File checkpointFile : checkpointFiles) {
       CheckpointReader reader = new CheckpointReader(checkpointFile);
-      maxMemTableId = reader.getMaxMemTableId();
-      checkpoints = reader.getCheckpoints();
-      if (!checkpoints.isEmpty()) {
+      List<Checkpoint> checkpoints = reader.getCheckpoints();
+      if (checkpoints.isEmpty()) {
+        continue;
+      }
+      readers.addFirst(reader);

Review Comment:
   **[P2] Avoid using a snapshot across a checkpoint version gap.** A failed 
deletion can leave `_0.checkpoint` behind, and a later legacy roll can delete 
`_1.checkpoint` even after failing to write the snapshot in `_2.checkpoint`. If 
`_1` recorded the flush of memtable 1 and `_2` has a readable create record for 
memtable 2 on the same unsealed TsFile, merging `_0` revives memtable 1. The 
stale mapping takes the file's recovery performer, so the durable insert for 
memtable 2 is skipped. I reproduced this with `_0` and `_2` present. The base 
recovery code replayed memtable 2. Please keep the readable increments after 
the gap and only use an older snapshot when the intervening checkpoint versions 
are available.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to