Caideyipi commented on code in PR #18531:
URL: https://github.com/apache/iotdb/pull/18531#discussion_r3871346115
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/util/LoadUtil.java:
##########
@@ -145,6 +152,19 @@ private static boolean loadTsFilesToActiveDir(
return true;
}
+ private static boolean isValidTsFile(final File file) {
Review Comment:
[P1] Apply this validation to the Pipe async path too
`IoTDBDataNodeReceiver.loadTsFileAsync` calls
`LoadUtil.loadFilesToActiveDir`, but `isValidTsFile` is only invoked from
`loadTsFilesToActiveDir`. The Pipe seal path therefore still transfers and
deletes a file with a bad suffix or invalid magic, then returns
`SUCCESS_STATUS`; the active loader fails later, after the sender has already
been acknowledged. Reuse the same validation in `loadFilesToActiveDir` before
`transferFilesToActiveDir`, and add coverage for this path.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/util/LoadUtil.java:
##########
@@ -118,6 +120,11 @@ private static boolean loadTsFilesToActiveDir(
return true;
}
+ // Validate before moving the source so ordinary or malformed files remain
in place.
+ if (!isValidTsFile(file)) {
Review Comment:
[P1] Validate the complete batch before moving any file
`loadTsFileAsyncToActiveDir` calls this method once per file. If a directory
contains a valid `*.tsfile` followed by an invalid file, the valid file has
already been copied/linked into the active-load directory before this call
returns `false` for the invalid one. `doAsyncLoad` then falls back to normal
analysis, but the first file remains queued; with `on-success='delete'`, its
source may already be deleted as well. This can cause partial ingestion and
unsafe retries. Validate all files before starting the transfer, or roll back
every transfer when a later validation fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
