JingsongLi commented on code in PR #11285:
URL: https://github.com/apache/paimon/pull/11285#discussion_r4657862207


##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,79 @@ public Plan readIncrementalDiff(Snapshot before) {
                 afterFiles);
     }
 
+    private FileStoreScan.Plan planForIncrementalDiff() {
+        return options.dataEvolutionEnabled()
+                ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+                : scan.plan();
+    }
+
+    private Plan toDataEvolutionIncrementalPlan(
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+            FileStoreScan.Plan afterPlan,
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+        Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new 
HashMap<>();
+        Map<Pair<BinaryRow, Integer>, RowRangeIndex> changedRanges = new 
HashMap<>();
+        RangeHelper<ManifestEntry> ranges =
+                new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+        for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part : 
afterFiles.entrySet()) {
+            for (Map.Entry<Integer, List<ManifestEntry>> bucket : 
part.getValue().entrySet()) {
+                Set<ManifestEntry> oldFiles =
+                        new HashSet<>(
+                                beforeFiles
+                                        .getOrDefault(part.getKey(), 
Collections.emptyMap())
+                                        .getOrDefault(bucket.getKey(), 
Collections.emptyList()));
+                List<Range> addedRanges = new ArrayList<>();
+                List<ManifestEntry> dependencies = new ArrayList<>();
+                for (List<ManifestEntry> group : 
ranges.mergeOverlappingRanges(bucket.getValue())) {
+                    boolean changed = false;
+                    for (ManifestEntry entry : group) {
+                        if (!oldFiles.contains(entry)) {
+                            addedRanges.add(entry.file().nonNullRowIdRange());
+                            changed = true;
+                        }
+                    }
+                    if (changed) {
+                        // Shared files still supply untouched columns and the 
current DV anchor.
+                        dependencies.addAll(group);

Review Comment:
   **[P2] Preserve file metadata for a single-column projection after a partial 
update.**
   
   On Spark 3.5.9, I inserted `(1, 10, 100)` into a Data Evolution table with 
columns `id, a, b`, then ran `UPDATE t SET b = 101 WHERE id = 1`. Selecting `a, 
__paimon_file_path, __paimon_row_index` from the latest snapshot returned `10` 
with the original file path and position `0`. DIFF over those same snapshots 
threw in `PaimonRecordReaderIterator.readBatch`.
   
   Retaining the `b`-only file routes the projection through the union reader 
even though the original file supplies all requested values. The union returns 
a plain `DataEvolutionIterator`, so the indexed wrapper has no file path or 
position to forward to Spark. Please determine the changed RowID ranges from 
the full inventory, then apply projection pruning to the selected dependencies 
while retaining any required deletion-vector anchor.



##########
paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionBatchScan.java:
##########
@@ -497,6 +498,15 @@ private boolean queryAuthEnabled() {
     }
 
     private Optional<GlobalIndexResult> evalGlobalIndex() {
+        CoreOptions options = table.coreOptions();
+        // Incremental endpoints are resolved by batchScan, not by the 
independent index scanner.
+        if (options.startupMode() == CoreOptions.StartupMode.INCREMENTAL) {

Review Comment:
   **[P2] Preserve caller-supplied scored results for DELTA reads.**
   
   I appended `100` in snapshot 1 and `200, 300` in snapshot 2 of a Data 
Evolution table. With `incremental-between=1,2` and a scored 
`GlobalIndexResult` selecting RowID 2, the base revision returned `300` with 
score `0.42` for both `incremental-between-scan-mode=delta` and `auto`. This 
guard now throws `IllegalArgumentException` for both modes before planning. The 
suggested `withRowRanges` replacement carries the row selection only.
   
   Please keep automatic index evaluation disabled for unpinned historical 
scans and scope the external-result rejection to the resolved DIFF path, so 
DELTA callers can continue to read rows with their scores.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to