Caideyipi commented on code in PR #18531:
URL: https://github.com/apache/iotdb/pull/18531#discussion_r3871346103
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/util/LoadUtil.java:
##########
@@ -118,6 +120,11 @@ private static boolean loadTsFilesToActiveDir(
return true;
}
+ // Validate before moving the source so ordinary or malformed files remain
in place.
+ if (!isValidTsFile(file)) {
Review Comment:
[P1] Validate the complete batch before moving any file
`loadTsFileAsyncToActiveDir` calls this method once per `*.tsfile`. If a
directory contains a valid `*.tsfile` followed by a malformed `*.tsfile`, the
valid file has already been copied/linked into the active-load directory before
this call returns `false` for the malformed one. `doAsyncLoad` then falls back
to normal analysis, but the first file remains queued; with
`on-success='delete'`, its source may already be deleted as well. This can
cause partial ingestion and unsafe retries. Validate all files before starting
the transfer, or roll back every transfer when a later validation fails.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/util/LoadUtil.java:
##########
@@ -145,6 +152,19 @@ private static boolean loadTsFilesToActiveDir(
return true;
}
+ private static boolean isValidTsFile(final File file) {
Review Comment:
[P1] Apply this validation to the Pipe async path too
`IoTDBDataNodeReceiver.loadTsFileAsync` calls
`LoadUtil.loadFilesToActiveDir`, but `isValidTsFile` is only invoked from
`loadTsFilesToActiveDir`. The Pipe seal path therefore still transfers and
deletes the main TsFile even when its magic is invalid, then returns
`SUCCESS_STATUS`; the active loader fails later, after the sender has already
been acknowledged. Identify the main `.tsfile` entry and apply the same
validation in `loadFilesToActiveDir` before `transferFilesToActiveDir` (while
preserving valid `.resource`/`.mods` sidecars), and add coverage for this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
