JingsongLi commented on code in PR #11285:
URL: https://github.com/apache/paimon/pull/11285#discussion_r4657862207
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -657,6 +668,79 @@ public Plan readIncrementalDiff(Snapshot before) {
afterFiles);
}
+ private FileStoreScan.Plan planForIncrementalDiff() {
+ return options.dataEvolutionEnabled()
+ ? ((DataEvolutionFileStoreScan) scan).planForIncrementalDiff()
+ : scan.plan();
+ }
+
+ private Plan toDataEvolutionIncrementalPlan(
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> beforeFiles,
+ FileStoreScan.Plan afterPlan,
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> afterFiles) {
+ Map<BinaryRow, Map<Integer, List<ManifestEntry>>> selected = new
HashMap<>();
+ Map<Pair<BinaryRow, Integer>, RowRangeIndex> changedRanges = new
HashMap<>();
+ RangeHelper<ManifestEntry> ranges =
+ new RangeHelper<>(entry -> entry.file().nonNullRowIdRange());
+ for (Map.Entry<BinaryRow, Map<Integer, List<ManifestEntry>>> part :
afterFiles.entrySet()) {
+ for (Map.Entry<Integer, List<ManifestEntry>> bucket :
part.getValue().entrySet()) {
+ Set<ManifestEntry> oldFiles =
+ new HashSet<>(
+ beforeFiles
+ .getOrDefault(part.getKey(),
Collections.emptyMap())
+ .getOrDefault(bucket.getKey(),
Collections.emptyList()));
+ List<Range> addedRanges = new ArrayList<>();
+ List<ManifestEntry> dependencies = new ArrayList<>();
+ for (List<ManifestEntry> group :
ranges.mergeOverlappingRanges(bucket.getValue())) {
+ boolean changed = false;
+ for (ManifestEntry entry : group) {
+ if (!oldFiles.contains(entry)) {
+ addedRanges.add(entry.file().nonNullRowIdRange());
+ changed = true;
+ }
+ }
+ if (changed) {
+ // Shared files still supply untouched columns and the
current DV anchor.
+ dependencies.addAll(group);
Review Comment:
**[P2] Preserve file metadata for a single-column projection after a partial
update.**
On Spark 3.5.9, I inserted `(1, 10, 100)` into a Data Evolution table with
columns `id, a, b`, then ran `UPDATE t SET b = 101 WHERE id = 1`. Selecting `a,
__paimon_file_path, __paimon_row_index` from the latest snapshot returned `10`
with the original file path and position `0`. DIFF over those same snapshots
threw in `PaimonRecordReaderIterator.readBatch`.
Retaining the `b`-only file routes the projection through the union reader
even though the original file supplies all requested values. The union returns
a plain `DataEvolutionIterator`, so the indexed wrapper has no file path or
position to forward to Spark. Please determine the changed RowID ranges from
the full inventory, then apply projection pruning to the selected dependencies
while retaining any required deletion-vector anchor.
##########
paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionBatchScan.java:
##########
@@ -497,6 +498,15 @@ private boolean queryAuthEnabled() {
}
private Optional<GlobalIndexResult> evalGlobalIndex() {
+ CoreOptions options = table.coreOptions();
+ // Incremental endpoints are resolved by batchScan, not by the
independent index scanner.
+ if (options.startupMode() == CoreOptions.StartupMode.INCREMENTAL) {
Review Comment:
**[P2] Preserve caller-supplied scored results for DELTA reads.**
I appended `100` in snapshot 1 and `200, 300` in snapshot 2 of a Data
Evolution table. With `incremental-between=1,2` and a scored
`GlobalIndexResult` selecting RowID 2, the base revision returned `300` with
score `0.42` for both `incremental-between-scan-mode=delta` and `auto`. This
guard now throws `IllegalArgumentException` for both modes before planning. The
suggested `withRowRanges` replacement carries the row selection only.
Please keep automatic index evaluation disabled for unpinned historical
scans and scope the external-result rejection to the resolved DIFF path, so
DELTA callers can continue to read rows with their scores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]