luoluoyuyu commented on PR #19031:
URL: https://github.com/apache/iotdb/pull/19031#issuecomment-6486856002

   Read the diff against `TsFileWriterManager` (fsync before records) and 
`recoverFromDisk` (truncate to `resumeOffset`), so the core fix looks sound. 
Issues introduced by this change:
   
   1. **Misleading WARN on empty progress logs.** `findTornTailOffset()` 
returns `0` for any file shorter than 4 bytes, and 
`readAllRecordsRepairingTornTail()` warns "dropped the trailing entry ... from 
offset 0" whenever the offset is `>= 0`. A normal snapshot that copies a staged 
file before its log gets its header, and recovery of a crashed task, now both 
report dropping an entry that never existed. Only warn when the offset is `> 0`.
   
   2. **One unreadable progress log now fails the whole region snapshot.** 
`discardUnrecordedStagedFiles()` lets the `IOException` from an invalid magic 
escape, so `snapshot()` returns false and the migration keeps retrying against 
a log the node cannot replay anyway. `recordedLengthOf()` previously swallowed 
it. Is failing the migration the intended blast radius, or should the 
unreadable task be left out of the snapshot?
   
   3. **The "log ⊆ data" invariant is only checked in tests.** 
`copyPrefixBytes()` breaks silently when `transferTo()` returns `0`, and 
`catchUpStagedFiles()` gives up after `MAX_STAGED_FILE_REPAIRS` without 
signaling anything, so `snapshot()` can return true with a log that describes 
bytes the copied file does not hold. Verifying max `physicalEnd` <= copied file 
length at the end of `copyTask` and returning false would make that fail loudly 
instead.
   
   4. **An empty `.progress` file is left behind** when a staged file is 
discarded, which is why `assertEveryCopiedLogFitsItsCopiedFile()` had to be 
relaxed to allow a log without data. Deleting the log too keeps the invariant 
strict. The deletion is also silent, which makes "why did this task replay from 
scratch after migration" hard to debug.
   
   5. **Behavior change outside the LOAD snapshot path.** `readAllRecords()` 
now returns empty for a 0-byte file, so 
`LoadTsFileCleaner.lastRecordedSearchIndex()` no longer hits the "read failed, 
leave it alone" branch and proceeds to delete/`dropTask`. A 0-byte log 
currently only exists inside the window in `ensureActivated()`; please confirm 
the cleaner is expected to release such a directory.
   
   6. Cost: hard link -> byte copy doubles staging disk usage during migration 
and makes snapshot time linear in pending LOAD size. Hard-linking files that 
can no longer be appended (terminal record present / resource NORMAL) and 
copying only the live ones may be worth it.
   
   Minor test notes: blocking inside `setChunkPayloadRefs()` keeps the writer 
thread waiting up to 30s under the manager lock; 
`config.setLoadTsFileWriteBufferSize(1024)` mutates the global `IoTDBConfig`; 
and nothing asserts that replaying after `restore()` leaves the snapshot bytes 
unchanged, which is the other half of the fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to