JingsongLi commented on code in PR #11258:
URL: https://github.com/apache/paimon/pull/11258#discussion_r4624214151


##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -573,8 +574,25 @@ private Plan toIncrementalPlan(
                     totalBuckets = beforeEntries.get(0).totalBuckets();
                 }
 
-                // deduplicate: remove entries common to both lists
-                deduplicate(beforeEntries, dataEntries);
+                Map<String, DeletionFile> beforeVectors =
+                        beforeDeletionFilesMap == null
+                                ? Collections.emptyMap()
+                                : beforeDeletionFilesMap.getOrDefault(
+                                        Pair.of(part, bucket), 
Collections.emptyMap());
+                Map<String, DeletionFile> afterVectors =
+                        afterDeletionFilesMap == null
+                                ? Collections.emptyMap()
+                                : afterDeletionFilesMap.getOrDefault(
+                                        Pair.of(part, bucket), 
Collections.emptyMap());
+                // A primary-key diff must read both versions of a shared file 
when its visible
+                // rows changed. Append diffs only read newly added files.
+                deduplicate(
+                        beforeEntries,
+                        dataEntries,
+                        file ->
+                                tableSchema.primaryKeys().isEmpty()
+                                        || Objects.equals(
+                                                beforeVectors.get(file), 
afterVectors.get(file)));

Review Comment:
   **[P2] Keep pruning files whose deletion bitmap is unchanged**
   
   `BucketedDvMaintainer.writeDeletionVectorsIndex()` rewrites all DVs in a 
bucket whenever one changes. `DeletionFile.equals()` includes the index path, 
so a bitmap copied unchanged into the new index compares unequal here. This can 
make a single-row delete trigger reads of every data file with a DV in the 
bucket, once for each snapshot.
   
   I reproduced this on this PR with a primary-key table `(k INT, v INT)` using 
`bucket=1`, `deletion-vectors.enabled=true`, and 
`compaction.small-file-ratio=0` to keep two separate files. Each commit waited 
for compaction.
   
   1. Write `(1,10), (2,20), (3,30)` in one commit and `(10,100), (11,110), 
(12,120)` in another.
   2. Delete keys `1` and `10` in one commit, then record the snapshot ID.
   3. Delete key `2` and run an `incremental-between` query from the recorded 
snapshot to the latest with `incremental-between-scan-mode=diff`.
   
   The diff correctly emits `DELETE (2,20)`, but its split contains two files 
on each side. I read both versions of the second file's DV and confirmed that 
the bitmaps are identical. The split only needs the first file on each side.
   
   Could we keep descriptor equality as the fast path, then compare bitmap 
contents when the descriptors differ and their cardinalities are equal or 
unknown? Different known cardinalities can keep the file without a bitmap read. 
The two-file case would also make a useful regression test for pruning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to