JingsongLi commented on code in PR #10964:
URL: https://github.com/apache/paimon/pull/10964#discussion_r4456264922
##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveCatalog.java:
##########
@@ -782,8 +817,20 @@ private List<Map<String, String>>
removePartitionsExistsInOtherBranches(
continue;
}
- branchTableForPartitionExistence(mainTable, branchName).newScan()
- .withPartitionsFilter(new
ArrayList<>(inputsToRemove)).listPartitions().stream()
+ FileStoreTable branchTable =
branchTableForPartitionExistence(mainTable, branchName);
+ InternalRowPartitionComputer partitionComputer =
+ new InternalRowPartitionComputer(
+ branchTable.coreOptions().partitionDefaultName(),
+
branchTable.rowType().project(branchTable.partitionKeys()),
+ branchTable.partitionKeys().toArray(new String[0]),
+ branchTable.coreOptions().legacyPartitionName());
+ DataTableScan scan = branchTable.newScan();
Review Comment:
[P2] Check marker ownership against current physical metadata
`newScan()` honors persisted read selectors, so it can miss a live literal
`.done` partition on the current branch. I extended
`testMarkerCleanupKeepsCurrentBranchRealDoneSuffixPartition` by first writing
`keep` as snapshot 1, then writing `20230101` and the real `20230101.done`, and
setting `scan.snapshot-id=1` with `catalog.alterTable` before the existing
explicit DROP of `20230101`. After removing the selector again, the latest
physical-data assertion still sees `keep` and `20230101.done`, but HMS contains
only `keep`: this new cleanup deleted the live partition's registration because
its ownership scan saw snapshot 1. The original explicit DROP did not touch
that unrequested HMS entry. Please inspect the physical branch's latest
snapshot partition metadata independently of query startup options, rather than
using a configured batch scan, and retain this real-partition collision
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]