Caideyipi commented on code in PR #18271:
URL: https://github.com/apache/iotdb/pull/18271#discussion_r3627205266
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/statement/crud/LoadTsFileStatement.java:
##########
@@ -195,6 +199,18 @@ private static void validateLoadSourcePath(final File
file) throws FileNotFoundE
Arrays.toString(allowedDirs)));
}
+ private static void validateNotLoadingInternalTsFile(final File file)
+ throws FileNotFoundException {
+ final Path sourcePath = canonicalPath(file);
+ for (final String dataDir :
IoTDBDescriptor.getInstance().getConfig().getLocalDataDirs()) {
Review Comment:
[P2] Hoist managed-directory canonicalization out of the recursive traversal
`findAllTsFile` calls this method for every file and directory it
encounters, and every call re-canonicalizes both `sequence` and `unsequence`
under every local data directory. Directory loading therefore performs `1 + 2 *
dataDirCount` canonical-path resolutions per entry, even though the managed
roots are invariant during one traversal.
On Windows/JDK 17, traversing 5,000 TsFiles with this PR took about 3.24 s
with 1 local data directory, 9.68 s with 4, and 37.31 s with 16, before LOAD
analysis or execution began. This makes large directory LOADs increasingly
expensive for the supported multi-directory configuration.
Please compute the canonical managed roots once at the start of
`processTsFile` and pass that snapshot through the recursion, or cache them
alongside the data-directory configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
