JustinBinber commented on code in PR #11285:
URL: https://github.com/apache/paimon/pull/11285#discussion_r4644787264
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
afterFiles);
}
+ private FileStoreScan.Plan planForIncrementalDiff() {
+ return options.dataEvolutionEnabled()
+ ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+ : scan.plan();
+ }
+
+ private Plan toDataEvolutionIncrementalPlan(
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+ FileStoreScan.Plan afterPlan,
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new
HashMap<>();
+ Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new
HashMap<>();
+ RangeHelper<ManifestEntry> ranges =
+ new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part :
afterFiles.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
part.getValue().entrySet()) {
+ Set<ManifestEntry> oldFiles =
+ new HashSet<>(
+ beforeFiles
+ .getOrDefault(part.getKey(),
Collections.emptyMap())
+ .getOrDefault(bucket.getKey(),
Collections.emptyList()));
+ List<Range> addedRanges = new ArrayList<>();
+ List<ManifestEntry> dependencies = new ArrayList<>();
+ for (List<ManifestEntry> group :
ranges.mergeOverlappingRanges(bucket.getValue())) {
+ boolean changed = false;
+ for (ManifestEntry entry : group) {
+ if (!oldFiles.contains(entry)) {
+ addedRanges.add(entry.file().nonNullRowIdRange());
+ changed = true;
+ }
+ }
+ if (changed) {
+ // Shared files still supply untouched columns and the
current DV anchor.
+ dependencies.addAll(group);
+ }
+ }
+ if (!addedRanges.isEmpty()) {
+ selected.computeIfAbsent(part.getKey(), ignored -> new
HashMap<>())
+ .put(bucket.getKey(), dependencies);
+ changedRanges.put(
+ Pair.of(part.getKey(), bucket.getKey()),
+ Range.sortAndMergeOverlap(addedRanges));
+ }
+ }
+ }
+ List<Split> splits = new ArrayList<>();
+ for (DataSplit split :
+ generateSplits(afterPlan.snapshot(), false, splitGenerator,
selected)) {
+ List<Range> splitRanges =
+ Range.sortAndMergeOverlap(
+ split.dataFiles().stream()
+ .map(DataFileMeta::nonNullRowIdRange)
+ .collect(Collectors.toList()));
+ List<Range> selectedRanges =
+ Range.and(
Review Comment:
Fixed in `07987830f3`. Changed ranges now build one RowRangeIndex per
partition/bucket. Each split looks up only its actual merged intervals, and
caller selections use the existing RowRangeIndex lookup too. Neither
intersection restarts a linear walk over the bucket's complete list. The
exact-range regression packs disjoint legal update groups, preserves the gap
and unchanged columns, and checks both contiguous and sparse caller selections;
the DV/BLOB and LIMIT regressions remain green.
A metadata-only JDK 8 probe uses the actual Range/RowRangeIndex classes,
gapped inputs, four small warmups, and three alternating-order repeats. Timings
below include building the changed-range index; outputs are checksum-checked
against the same expected ranges. No other JVM/build ran during the probe.
| Changed ranges | Old linear intersection median (ms) | Indexed median (ms)
|
| ---: | ---: | ---: |
| 25,000 | 333.43 | 6.84 |
| 50,000 | 1,354.68 | 7.71 |
| 100,000 | 5,367.74 | 12.48 |
With an additional sparse caller selection, 100,000 ranges changes from
10,151.40 ms to 26.11 ms. This is only the intersection stage, not full
snapshot planning, SQL or distributed performance. The current normal Maven
Core reactor passes all 44 focused cases; current-source Spark SQL controls
also pass. Fresh upstream CI is automatic and pending for this head.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
afterFiles);
}
+ private FileStoreScan.Plan planForIncrementalDiff() {
+ return options.dataEvolutionEnabled()
+ ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+ : scan.plan();
+ }
+
+ private Plan toDataEvolutionIncrementalPlan(
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+ FileStoreScan.Plan afterPlan,
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new
HashMap<>();
+ Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new
HashMap<>();
+ RangeHelper<ManifestEntry> ranges =
+ new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part :
afterFiles.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
part.getValue().entrySet()) {
+ Set<ManifestEntry> oldFiles =
+ new HashSet<>(
+ beforeFiles
+ .getOrDefault(part.getKey(),
Collections.emptyMap())
+ .getOrDefault(bucket.getKey(),
Collections.emptyList()));
+ List<Range> addedRanges = new ArrayList<>();
+ List<ManifestEntry> dependencies = new ArrayList<>();
+ for (List<ManifestEntry> group :
ranges.mergeOverlappingRanges(bucket.getValue())) {
+ boolean changed = false;
+ for (ManifestEntry entry : group) {
+ if (!oldFiles.contains(entry)) {
+ addedRanges.add(entry.file().nonNullRowIdRange());
+ changed = true;
+ }
+ }
+ if (changed) {
+ // Shared files still supply untouched columns and the
current DV anchor.
+ dependencies.addAll(group);
+ }
+ }
+ if (!addedRanges.isEmpty()) {
+ selected.computeIfAbsent(part.getKey(), ignored -> new
HashMap<>())
+ .put(bucket.getKey(), dependencies);
+ changedRanges.put(
+ Pair.of(part.getKey(), bucket.getKey()),
+ Range.sortAndMergeOverlap(addedRanges));
+ }
+ }
+ }
+ List<Split> splits = new ArrayList<>();
+ for (DataSplit split :
+ generateSplits(afterPlan.snapshot(), false, splitGenerator,
selected)) {
+ List<Range> splitRanges =
+ Range.sortAndMergeOverlap(
+ split.dataFiles().stream()
+ .map(DataFileMeta::nonNullRowIdRange)
+ .collect(Collectors.toList()));
+ List<Range> selectedRanges =
+ Range.and(
+ splitRanges,
+ changedRanges.get(Pair.of(split.partition(),
split.bucket())));
+ if (rowRangeIndex != null) {
+ selectedRanges = Range.and(selectedRanges,
rowRangeIndex.ranges());
+ }
+ if (!selectedRanges.isEmpty()) {
+ splits.add(new IndexedSplit(split, selectedRanges, null));
Review Comment:
Reproduced and fixed in `07987830f3`. When the underlying iterator provides
physical file metadata, a private adapter preserves both FileRecordIterator and
ScoreRecordIterator, including across filter and transform. Path and physical
position delegate to the actual file iterator; logical RowID and score keep
their existing mapping. Release and reader-close ownership are unchanged.
Non-file delegates retain their previous behavior.
Retaining the score interface matters because lateral vector-search row
fetching deliberately supplies unscored IDs and later restores per-query
scores. Regressions verify logical RowIDs 7/10/11 versus physical positions
1/4/5, the exact file path, and NaN temporary scores after
filtering/projection. Real Spark DIFF metadata projection also checks deletion
holes and filtered physical indexes 2/5. The lateral score-plus-data query
passes for both DOUBLE and FLOAT query vectors with actual final scores 1.0.
All 44 Core cases and eight selected Spark cases pass, including the five
compaction race controls, with current Core class locations verified. Fresh
upstream CI is pending for this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]