JustinBinber opened a new pull request, #11285:
URL: https://github.com/apache/paimon/pull/11285

   ### Purpose
   
   Data Evolution batch `DIFF` scans currently produce an `IncrementalSplit` 
that the DE reader casts to `IndexedSplit`, so nonempty snapshot and tag 
differences fail with `ClassCastException`.
   
   Reading only the newly added column files would also lose untouched columns 
and the shared deletion-vector anchor. This change identifies changed RowID 
ranges from complete manifest-entry identities, retains their complete 
end-snapshot dependency groups, and reuses existing range-selected 
`IndexedSplit` reads. Projection, data-statistics filters and LIMIT do not 
prune either metadata inventory before the difference is computed; the caller's 
scan settings are restored afterward. Partition pruning remains in place. 
Deletion vectors come from the end snapshot.
   
   `IndexedSplit` no longer advertises an exact visible row count when it has 
deletion vectors, preventing LIMIT from stopping at an entirely deleted split.
   
   The documented contract is file incremental: return visible end-snapshot 
rows in ranges covered by added file entries. Unchanged columns are 
reconstructed, compaction may repeat rows, and deletion-only changes produce no 
positive rows. This is not row-level CDC and does not introduce streaming 
metadata planning. Ordinary append and primary-key DIFF paths remain unchanged.
   
   This is an independent core correctness fix discovered while investigating 
#11156; it does not include that PR's memory optimization.
   
   ### Tests
   
   Normal Maven reactor verification on JDK 8 / Scala 2.12 / Spark 3.5.9, 
without `fast-build`: all 26 required modules built successfully, 17 Core JUnit 
tests and the new Spark SQL test passed.
   
   ```shell
   mvn -pl paimon-spark/paimon-spark-ut -am -Pflink1,spark3 \
     -Dflink.forkCount=1 \
     -Dtest=DataEvolutionIncrementalDiffTest,IndexedSplitTest \
     
-DwildcardSuites=org.apache.paimon.spark.commands.DataEvolutionIncrementalDiffTest
 \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false package
   ```
   
   Coverage includes implicit tag differences, duplicate/NULL rows, multiple 
partial-column updates, unchanged-column projection, predicates, end-snapshot 
deletion vectors on shared anchors, a dedicated BLOB subrange inside a wider 
normal range, serialized splits, deletion-only/empty differences, and LIMIT 
across a fully deleted split. Ordinary append and primary-key reads are checked 
as controls. The Spark test executes real DELETE and UPDATE SQL before reading 
DIFF and checks full rows, projection, predicates, ordering and LIMIT.
   
   The unchanged baseline fails the seven DE integration cases while the 
ordinary-table control passes. Removing only the deletion-vector row-count fix 
reproduces an empty LIMIT result where a later live row should be returned.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to