luoluoyuyu commented on PR #19031: URL: https://github.com/apache/iotdb/pull/19031#issuecomment-6486856002
Read the diff against `TsFileWriterManager` (fsync before records) and `recoverFromDisk` (truncate to `resumeOffset`), so the core fix looks sound. Issues introduced by this change: 1. **Misleading WARN on empty progress logs.** `findTornTailOffset()` returns `0` for any file shorter than 4 bytes, and `readAllRecordsRepairingTornTail()` warns "dropped the trailing entry ... from offset 0" whenever the offset is `>= 0`. A normal snapshot that copies a staged file before its log gets its header, and recovery of a crashed task, now both report dropping an entry that never existed. Only warn when the offset is `> 0`. 2. **One unreadable progress log now fails the whole region snapshot.** `discardUnrecordedStagedFiles()` lets the `IOException` from an invalid magic escape, so `snapshot()` returns false and the migration keeps retrying against a log the node cannot replay anyway. `recordedLengthOf()` previously swallowed it. Is failing the migration the intended blast radius, or should the unreadable task be left out of the snapshot? 3. **The "log ⊆ data" invariant is only checked in tests.** `copyPrefixBytes()` breaks silently when `transferTo()` returns `0`, and `catchUpStagedFiles()` gives up after `MAX_STAGED_FILE_REPAIRS` without signaling anything, so `snapshot()` can return true with a log that describes bytes the copied file does not hold. Verifying max `physicalEnd` <= copied file length at the end of `copyTask` and returning false would make that fail loudly instead. 4. **An empty `.progress` file is left behind** when a staged file is discarded, which is why `assertEveryCopiedLogFitsItsCopiedFile()` had to be relaxed to allow a log without data. Deleting the log too keeps the invariant strict. The deletion is also silent, which makes "why did this task replay from scratch after migration" hard to debug. 5. **Behavior change outside the LOAD snapshot path.** `readAllRecords()` now returns empty for a 0-byte file, so `LoadTsFileCleaner.lastRecordedSearchIndex()` no longer hits the "read failed, leave it alone" branch and proceeds to delete/`dropTask`. A 0-byte log currently only exists inside the window in `ensureActivated()`; please confirm the cleaner is expected to release such a directory. 6. Cost: hard link -> byte copy doubles staging disk usage during migration and makes snapshot time linear in pending LOAD size. Hard-linking files that can no longer be appended (terminal record present / resource NORMAL) and copying only the live ones may be worth it. Minor test notes: blocking inside `setChunkPayloadRefs()` keeps the writer thread waiting up to 30s under the manager lock; `config.setLoadTsFileWriteBufferSize(1024)` mutates the global `IoTDBConfig`; and nothing asserts that replaying after `restore()` leaves the snapshot bytes unchanged, which is the other half of the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
