JustinBinber opened a new pull request, #11285:
URL: https://github.com/apache/paimon/pull/11285
### Purpose
Data Evolution batch `DIFF` scans currently produce an `IncrementalSplit`
that the DE reader casts to `IndexedSplit`, so nonempty snapshot and tag
differences fail with `ClassCastException`.
Reading only the newly added column files would also lose untouched columns
and the shared deletion-vector anchor. This change identifies changed RowID
ranges from complete manifest-entry identities, retains their complete
end-snapshot dependency groups, and reuses existing range-selected
`IndexedSplit` reads. Projection, data-statistics filters and LIMIT do not
prune either metadata inventory before the difference is computed; the caller's
scan settings are restored afterward. Partition pruning remains in place.
Deletion vectors come from the end snapshot.
`IndexedSplit` no longer advertises an exact visible row count when it has
deletion vectors, preventing LIMIT from stopping at an entirely deleted split.
The documented contract is file incremental: return visible end-snapshot
rows in ranges covered by added file entries. Unchanged columns are
reconstructed, compaction may repeat rows, and deletion-only changes produce no
positive rows. This is not row-level CDC and does not introduce streaming
metadata planning. Ordinary append and primary-key DIFF paths remain unchanged.
This is an independent core correctness fix discovered while investigating
#11156; it does not include that PR's memory optimization.
### Tests
Normal Maven reactor verification on JDK 8 / Scala 2.12 / Spark 3.5.9,
without `fast-build`: all 26 required modules built successfully, 17 Core JUnit
tests and the new Spark SQL test passed.
```shell
mvn -pl paimon-spark/paimon-spark-ut -am -Pflink1,spark3 \
-Dflink.forkCount=1 \
-Dtest=DataEvolutionIncrementalDiffTest,IndexedSplitTest \
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionIncrementalDiffTest
\
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false package
```
Coverage includes implicit tag differences, duplicate/NULL rows, multiple
partial-column updates, unchanged-column projection, predicates, end-snapshot
deletion vectors on shared anchors, a dedicated BLOB subrange inside a wider
normal range, serialized splits, deletion-only/empty differences, and LIMIT
across a fully deleted split. Ordinary append and primary-key reads are checked
as controls. The Spark test executes real DELETE and UPDATE SQL before reading
DIFF and checks full rows, projection, predicates, ordering and LIMIT.
The unchanged baseline fails the seven DE integration cases while the
ordinary-table control passes. Removing only the deletion-vector row-count fix
reproduces an empty LIMIT result where a later live row should be returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]