luoluoyuyu commented on PR #19073:
URL: https://github.com/apache/iotdb/pull/19073#issuecomment-6573282705

   Thanks — turning a silent partial load into a failure is the right 
direction. A few concerns before merge:
   
   1. **The checkpoint is a per-series time watermark.** `recordProgress` 
advances *every* measurement in a tablet to the tablet's last row timestamp, 
and `readDeviceMeasurementsInOrder` treats `emitted >= statistics.getEndTime()` 
as "fully loaded". That only holds if each series' timestamps are 
non-decreasing across chunk groups, which TsFile writers don't enforce. If a 
later group holds an earlier timestamp, those points are skipped without 
recording a failure → `SUCCESS` + `on-success=delete` still deletes the source, 
i.e. the same bug class. Suggest advancing only to the last row where that 
measurement really has a value (nulls in aligned tablets over-advance it), plus 
a test with a non-monotonic file.
   
   2. **No best-effort path.** One unreadable measurement now makes the whole 
file permanently unloadable, even when the corrupt region contains no data. 
Worth an explicit decision (or an opt-in flag).
   
   3. **Upstream retry heuristics key off message text** 
(`ActiveLoadFailedMessageHandler.isExceptionMessageShouldRetry`). The surfaced 
message is the generic `"Failed to iterate tablets while loading TsFile."`, so 
even transient IO errors get moved to the fail dir. Including 
device/measurement and the cause (via a `DataNodeQueryMessages` constant) would 
make this diagnosable and retryable.
   
   4. **Task count**: one `QueryTask` per distinct checkpoint means one 
`TsFileReader` open per task; wide devices can turn fallback into thousands of 
file opens, and the split path can emit O(M) WARNs each printing the full 
measurement list.
   
   5. Minor: a split task whose halves are all filtered out by `emitted >= 
endTime` loses the failure entirely; only the first failure is reported; and 
`waitUntilClusterAnswers()` returning `true` unconditionally weakens the IT 
readiness check (unrelated to this fix).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to