hongzhi-gao opened a new pull request, #18872:
URL: https://github.com/apache/iotdb/pull/18872

   ## Description
   
   A TsFileProcessor can remain open after its working memtable is detached for 
flush. Tree and table deletions currently dereference that null memtable while 
logging to WAL. Handling this state must also preserve deletion replay and 
prevent a deletion registered during flush completion from being acknowledged 
without being persisted.
   
   ### Deletion logging and flush ordering
   
   - Associate a deletion with the working memtable, or the last pending 
flushing memtable when no working memtable exists. Use the existing 
`MEMTABLE_NOT_EXIST` sentinel only when neither remains, while retaining the 
WAL entry for IoTConsensus replication.
   - Hold the existing flush-query read lock through ID selection and WAL 
enqueue, before the selected memtable can be released and checkpointed.
   - Persist queued modifications and release the flushed memtable under one 
write-lock critical section. This closes the interval where a deletion could be 
registered after the modifications scan and then lost during release. Preserve 
the existing deque-monitor-to-write-lock order.
   
   ### Shared-file WAL recovery
   
   Map every unflushed memtable ID in a TsFile to the same recovery performer. 
Deduplicate performer lifecycle calls so the shared file is started, finalized, 
and closed once. The WAL and RPC formats remain unchanged.
   
   ### Validation
   
   ```bash
   mvn spotless:apply -pl iotdb-core/datanode
   mvn spotless:check -o -pl iotdb-core/datanode
   mvn test -o -pl iotdb-core/datanode -am 
-Dtest=TsFileProcessorTest,WALRecoverManagerTest,UnsealedTsFileRecoverPerformerTest
 -Dsurefire.failIfNoSpecifiedTests=false
   git diff --check
   ```
   
   All 36 tests passed, with no failures, errors, or skipped tests. The new 
tree/table WAL tests reproduce the NPE on the original implementation. The 
synchronized flush-completion regression fails before the locking fix because 
the persisted modifications are empty, and passes after the fix. The recovery 
regression uses real checkpoint/WAL files and checks both deleted and retained 
values after recovery.
   
   English and Chinese reactor test compilation compiled DataNode and the 
preceding modules successfully:
   
   ```bash
   mvn test-compile -o -DskipTests
   mvn test-compile -o -DskipTests -P with-zh-locale
   ```
   
   Both offline runs initially stopped at `distribution` due to uncached 
snapshot ZIP dependencies. Fetching those dependencies and resuming with `mvn 
test-compile -DskipTests -rf :iotdb-distribution` passed for distribution and 
all remaining modules.
   
   Cluster tests, process-crash fault injection, and disk-I/O fault injection 
were not run.
   
   <hr>
   
   This PR has:
   
   - [x] been self-reviewed, including concurrent deletion/flush and recovery 
ordering.
   - [x] added comments explaining the intent and lock ordering.
   - [x] added regression tests for deletion WAL logging, flush completion, and 
shared-file recovery.
   
   <hr>
   
   ##### Key changed classes
   
   - `TsFileProcessor`
   - `WALNodeRecoverTask`
   - `TsFileProcessorTest`
   - `WALRecoverManagerTest`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to