JingsongLi commented on code in PR #11285:
URL: https://github.com/apache/paimon/pull/11285#discussion_r4643166325
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
afterFiles);
}
+ private FileStoreScan.Plan planForIncrementalDiff() {
+ return options.dataEvolutionEnabled()
+ ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+ : scan.plan();
+ }
+
+ private Plan toDataEvolutionIncrementalPlan(
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+ FileStoreScan.Plan afterPlan,
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new
HashMap<>();
+ Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new
HashMap<>();
+ RangeHelper<ManifestEntry> ranges =
+ new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part :
afterFiles.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
part.getValue().entrySet()) {
+ Set<ManifestEntry> oldFiles =
+ new HashSet<>(
+ beforeFiles
+ .getOrDefault(part.getKey(),
Collections.emptyMap())
+ .getOrDefault(bucket.getKey(),
Collections.emptyList()));
+ List<Range> addedRanges = new ArrayList<>();
+ List<ManifestEntry> dependencies = new ArrayList<>();
+ for (List<ManifestEntry> group :
ranges.mergeOverlappingRanges(bucket.getValue())) {
+ boolean changed = false;
+ for (ManifestEntry entry : group) {
+ if (!oldFiles.contains(entry)) {
+ addedRanges.add(entry.file().nonNullRowIdRange());
+ changed = true;
+ }
+ }
+ if (changed) {
+ // Shared files still supply untouched columns and the
current DV anchor.
+ dependencies.addAll(group);
+ }
+ }
+ if (!addedRanges.isEmpty()) {
+ selected.computeIfAbsent(part.getKey(), ignored -> new
HashMap<>())
+ .put(bucket.getKey(), dependencies);
+ changedRanges.put(
+ Pair.of(part.getKey(), bucket.getKey()),
+ Range.sortAndMergeOverlap(addedRanges));
+ }
+ }
+ }
+ List<Split> splits = new ArrayList<>();
+ for (DataSplit split :
+ generateSplits(afterPlan.snapshot(), false, splitGenerator,
selected)) {
+ List<Range> splitRanges =
+ Range.sortAndMergeOverlap(
+ split.dataFiles().stream()
+ .map(DataFileMeta::nonNullRowIdRange)
+ .collect(Collectors.toList()));
+ List<Range> selectedRanges =
+ Range.and(
Review Comment:
**[P2] Avoid scanning every changed RowID range for every split.**
`Range.and` starts the right cursor at zero on every call. Adjacent appended
file ranges remain separate with the current `sortAndMergeOverlap` call. In a
Data Evolution append table with a large partition and one normal file per
split, 100,000 changed ranges cause about five billion range visits before
LIMIT can prune splits. A metadata-only JDK 8 probe using the actual range
classes took 0.43 s, 1.97 s, and 7.95 s for 25k, 50k, and 100k ranges. Build a
`RowRangeIndex` once per partition and bucket, then look up the ranges that
intersect each split interval. The same lookup can reuse the existing index for
the caller's requested RowIDs.
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,74 @@ public Plan readIncrementalDiff(Snapshot before) {
afterFiles);
}
+ private FileStoreScan.Plan planForIncrementalDiff() {
+ return options.dataEvolutionEnabled()
+ ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+ : scan.plan();
+ }
+
+ private Plan toDataEvolutionIncrementalPlan(
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+ FileStoreScan.Plan afterPlan,
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new
HashMap<>();
+ Map<Pair<BinaryRow, Integer>, List<Range>> changedRanges = new
HashMap<>();
+ RangeHelper<ManifestEntry> ranges =
+ new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part :
afterFiles.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
part.getValue().entrySet()) {
+ Set<ManifestEntry> oldFiles =
+ new HashSet<>(
+ beforeFiles
+ .getOrDefault(part.getKey(),
Collections.emptyMap())
+ .getOrDefault(bucket.getKey(),
Collections.emptyList()));
+ List<Range> addedRanges = new ArrayList<>();
+ List<ManifestEntry> dependencies = new ArrayList<>();
+ for (List<ManifestEntry> group :
ranges.mergeOverlappingRanges(bucket.getValue())) {
+ boolean changed = false;
+ for (ManifestEntry entry : group) {
+ if (!oldFiles.contains(entry)) {
+ addedRanges.add(entry.file().nonNullRowIdRange());
+ changed = true;
+ }
+ }
+ if (changed) {
+ // Shared files still supply untouched columns and the
current DV anchor.
+ dependencies.addAll(group);
+ }
+ }
+ if (!addedRanges.isEmpty()) {
+ selected.computeIfAbsent(part.getKey(), ignored -> new
HashMap<>())
+ .put(bucket.getKey(), dependencies);
+ changedRanges.put(
+ Pair.of(part.getKey(), bucket.getKey()),
+ Range.sortAndMergeOverlap(addedRanges));
+ }
+ }
+ }
+ List<Split> splits = new ArrayList<>();
+ for (DataSplit split :
+ generateSplits(afterPlan.snapshot(), false, splitGenerator,
selected)) {
+ List<Range> splitRanges =
+ Range.sortAndMergeOverlap(
+ split.dataFiles().stream()
+ .map(DataFileMeta::nonNullRowIdRange)
+ .collect(Collectors.toList()));
+ List<Range> selectedRanges =
+ Range.and(
+ splitRanges,
+ changedRanges.get(Pair.of(split.partition(),
split.bucket())));
+ if (rowRangeIndex != null) {
+ selectedRanges = Range.and(selectedRanges,
rowRangeIndex.ranges());
+ }
+ if (!selectedRanges.isEmpty()) {
+ splits.add(new IndexedSplit(split, selectedRanges, null));
Review Comment:
**[P2] Preserve file metadata when reading DIFF splits in Spark.** I
inserted IDs 1 and 2 in separate snapshots of a plain Data Evolution append
table. Spark 3.5.9 successfully read `id, __paimon_file_path,
__paimon_row_index` for ID 2 from the latest snapshot. The same projection on
DIFF 1→2 failed in `PaimonRecordReaderIterator.readBatch` because it received a
`ScoreRecordIterator` where it expected a `FileRecordIterator`.
`IndexedSplitRecordReader` drops the delegate's file path and returned
position. Please preserve those values for the unscored DIFF range reader and
add a metadata projection regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]