JingsongLi commented on code in PR #11285:
URL: https://github.com/apache/paimon/pull/11285#discussion_r4643166325


##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
                 afterFiles);
     }
 
+    private FileStoreScan.Plan planForIncrementalDiff() {
+        return options.dataEvolutionEnabled()
+                ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+                : scan.plan();
+    }
+
+    private Plan toDataEvolutionIncrementalPlan(
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+            FileStoreScan.Plan afterPlan,
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+        Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new 
HashMap<>();
+        Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new 
HashMap<>();
+        RangeHelper<ManifestEntry> ranges =
+                new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+        for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part : 
afterFiles.entrySet()) {
+            for (Map.Entry<Integer, List<ManifestEntry>> bucket : 
part.getValue().entrySet()) {
+                Set<ManifestEntry> oldFiles =
+                        new HashSet<>(
+                                beforeFiles
+                                        .getOrDefault(part.getKey(), 
Collections.emptyMap())
+                                        .getOrDefault(bucket.getKey(), 
Collections.emptyList()));
+                List<Range> addedRanges = new ArrayList<>();
+                List<ManifestEntry> dependencies = new ArrayList<>();
+                for (List<ManifestEntry> group : 
ranges.mergeOverlappingRanges(bucket.getValue())) {
+                    boolean changed = false;
+                    for (ManifestEntry entry : group) {
+                        if (!oldFiles.contains(entry)) {
+                            addedRanges.add(entry.file().nonNullRowIdRange());
+                            changed = true;
+                        }
+                    }
+                    if (changed) {
+                        // Shared files still supply untouched columns and the 
current DV anchor.
+                        dependencies.addAll(group);
+                    }
+                }
+                if (!addedRanges.isEmpty()) {
+                    selected.computeIfAbsent(part.getKey(), ignored -> new 
HashMap<>())
+                            .put(bucket.getKey(), dependencies);
+                    changedRanges.put(
+                            Pair.of(part.getKey(), bucket.getKey()),
+                            Range.sortAndMergeOverlap(addedRanges));
+                }
+            }
+        }
+        List<Split> splits = new ArrayList<>();
+        for (DataSplit split :
+                generateSplits(afterPlan.snapshot(), false, splitGenerator, 
selected)) {
+            List<Range> splitRanges =
+                    Range.sortAndMergeOverlap(
+                            split.dataFiles().stream()
+                                    .map(DataFileMeta::nonNullRowIdRange)
+                                    .collect(Collectors.toList()));
+            List<Range> selectedRanges =
+                    Range.and(

Review Comment:
   **[P2] Avoid scanning every changed RowID range for every split.** 
`Range.and` starts the right cursor at zero on every call. Adjacent appended 
file ranges remain separate with the current `sortAndMergeOverlap` call. In a 
Data Evolution append table with a large partition and one normal file per 
split, 100,000 changed ranges cause about five billion range visits before 
LIMIT can prune splits. A metadata-only JDK 8 probe using the actual range 
classes took 0.43 s, 1.97 s, and 7.95 s for 25k, 50k, and 100k ranges. Build a 
`RowRangeIndex` once per partition and bucket, then look up the ranges that 
intersect each split interval. The same lookup can reuse the existing index for 
the caller's requested RowIDs.



##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
                 afterFiles);
     }
 
+    private FileStoreScan.Plan planForIncrementalDiff() {
+        return options.dataEvolutionEnabled()
+                ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+                : scan.plan();
+    }
+
+    private Plan toDataEvolutionIncrementalPlan(
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+            FileStoreScan.Plan afterPlan,
+            Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+        Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new 
HashMap<>();
+        Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new 
HashMap<>();
+        RangeHelper<ManifestEntry> ranges =
+                new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+        for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part : 
afterFiles.entrySet()) {
+            for (Map.Entry<Integer, List<ManifestEntry>> bucket : 
part.getValue().entrySet()) {
+                Set<ManifestEntry> oldFiles =
+                        new HashSet<>(
+                                beforeFiles
+                                        .getOrDefault(part.getKey(), 
Collections.emptyMap())
+                                        .getOrDefault(bucket.getKey(), 
Collections.emptyList()));
+                List<Range> addedRanges = new ArrayList<>();
+                List<ManifestEntry> dependencies = new ArrayList<>();
+                for (List<ManifestEntry> group : 
ranges.mergeOverlappingRanges(bucket.getValue())) {
+                    boolean changed = false;
+                    for (ManifestEntry entry : group) {
+                        if (!oldFiles.contains(entry)) {
+                            addedRanges.add(entry.file().nonNullRowIdRange());
+                            changed = true;
+                        }
+                    }
+                    if (changed) {
+                        // Shared files still supply untouched columns and the 
current DV anchor.
+                        dependencies.addAll(group);
+                    }
+                }
+                if (!addedRanges.isEmpty()) {
+                    selected.computeIfAbsent(part.getKey(), ignored -> new 
HashMap<>())
+                            .put(bucket.getKey(), dependencies);
+                    changedRanges.put(
+                            Pair.of(part.getKey(), bucket.getKey()),
+                            Range.sortAndMergeOverlap(addedRanges));
+                }
+            }
+        }
+        List<Split> splits = new ArrayList<>();
+        for (DataSplit split :
+                generateSplits(afterPlan.snapshot(), false, splitGenerator, 
selected)) {
+            List<Range> splitRanges =
+                    Range.sortAndMergeOverlap(
+                            split.dataFiles().stream()
+                                    .map(DataFileMeta::nonNullRowIdRange)
+                                    .collect(Collectors.toList()));
+            List<Range> selectedRanges =
+                    Range.and(
+                            splitRanges,
+                            changedRanges.get(Pair.of(split.partition(), 
split.bucket())));
+            if (rowRangeIndex != null) {
+                selectedRanges = Range.and(selectedRanges, 
rowRangeIndex.ranges());
+            }
+            if (!selectedRanges.isEmpty()) {
+                splits.add(new IndexedSplit(split, selectedRanges, null));

Review Comment:
   **[P2] Preserve file metadata when reading DIFF splits in Spark.** I 
inserted IDs 1 and 2 in separate snapshots of a plain Data Evolution append 
table. Spark 3.5.9 successfully read `id, __paimon_file_path, 
__paimon_row_index` for ID 2 from the latest snapshot. The same projection on 
DIFF 1→2 failed in `PaimonRecordReaderIterator.readBatch` because it received a 
`ScoreRecordIterator` where it expected a `FileRecordIterator`. 
`IndexedSplitRecordReader` drops the delegate's file path and returned 
position. Please preserve those values for the unscored DIFF range reader and 
add a metadata projection regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to