zhangyue19921010 commented on code in PR #11258:
URL: https://github.com/apache/paimon/pull/11258#discussion_r4624263637


##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -573,8 +574,25 @@ private Plan toIncrementalPlan(
                     totalBuckets = beforeEntries.get(0).totalBuckets();
                 }
 
-                // deduplicate: remove entries common to both lists
-                deduplicate(beforeEntries, dataEntries);
+                Map<String, DeletionFile> beforeVectors =
+                        beforeDeletionFilesMap == null
+                                ? Collections.emptyMap()
+                                : beforeDeletionFilesMap.getOrDefault(
+                                        Pair.of(part, bucket), 
Collections.emptyMap());
+                Map<String, DeletionFile> afterVectors =
+                        afterDeletionFilesMap == null
+                                ? Collections.emptyMap()
+                                : afterDeletionFilesMap.getOrDefault(
+                                        Pair.of(part, bucket), 
Collections.emptyMap());
+                // A primary-key diff must read both versions of a shared file 
when its visible
+                // rows changed. Append diffs only read newly added files.
+                deduplicate(
+                        beforeEntries,
+                        dataEntries,
+                        file ->
+                                tableSchema.primaryKeys().isEmpty()
+                                        || Objects.equals(
+                                                beforeVectors.get(file), 
afterVectors.get(file)));

Review Comment:
   Done in dd73c4a: descriptor equality stays the fast path, different known 
cardinalities keep the file without a read, and otherwise the two bitmaps are 
read and compared, so a vector copied unchanged into the rewritten index keeps 
its file pruned. Added your two-file case as a regression test; it saw two 
files per side before this change and one now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to