Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278222805
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath,
String excludedChildPat
}
deleteDir(dir);
}
- List<FileEntry> files =
getFileSystem().listFiles(Location.of(targetPath));
+ // Storage implementations may treat listFiles arguments as globs.
Overwrite must enumerate
+ // the literal target, or a table name such as table_{prod} can
leave old rows behind.
+ List<FileEntry> files = new ArrayList<>();
+ try (FileIterator iterator =
getFileSystem().list(Location.of(targetPath))) {
+ while (iterator.hasNext()) {
+ FileEntry file = iterator.next();
+ if (!file.isDirectory()) {
Review Comment:
Fixed in d715bf7dede. Both cleanup loops now exclude the staging root and
every descendant, using the existing filesystem-aware equality and
slash-boundary containment helpers. The directory loop also excludes nested
staging directories if a provider lists them directly. Literal target
enumeration and existing glob discovery behavior remain unchanged.
Added a transaction regression using real AzureFileSystem enumeration and
copy-then-delete rename with an in-memory paginated AzureObjStorage. Before the
fix it fails with FileNotFoundException for the staged source, including an
otherwise empty table. The test covers 12 combinations of literal target names,
empty/nonempty old data and directory-listing shapes. It checks that new data
is published, other-query staging and adjacent tables survive, and
direct/nested old files plus .doris_staging_backup are cleaned up. No live
OneLake endpoint was used.
Validation: 604 scoped Hive/Azure/HDFS/shared-HMS tests passed with zero
failures, errors or skips; FE Checkstyle passed in 21 reactor modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
