JingsongLi commented on code in PR #1422:
URL: https://github.com/apache/paimon-rust/pull/1422#discussion_r4601208603


##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -551,7 +571,12 @@ impl DataEvolutionReader {
                     if remaining == Some(0) {
                         break 'splits;
                     }
-                    if is_raw_convertible(&files) {
+                    // Change events are physical file changes. Files of a 
streaming
+                    // split that are not column slices of one row range are
+                    // independent changes and are read as they are stored.
+                    if is_raw_convertible(&files)
+                        || (split.is_streaming() && 
!is_data_evolution_column_slice_group(&files))

Review Comment:
   [P1] Apply physical-event semantics before scan pruning
   
   Selecting raw events only here is too late: TableScan has already applied 
merged-row statistics and read-field pruning. On this PR's own two inserts 
(values 1..4) → compaction → value-only update to 50 fixture, the full 
timestamp window returns 8 physical events. With `with_projection(&["id"])`, 
its scan drops the update-only file and returns 4 rows; with `with_filter(value 
= 1)`, latest-provider stats (min=max=50) drop the entire group and return zero 
instead of the original `(1,1)` event. Both are public-API reproductions on 
this exact head. Reading the original unfiltered/full-width plan through the 
same projected or filtered reader returns 8 or 1 respectively.
   
   Please classify these physical-event groups before planner statistics/field 
pruning and use event-safe per-file pruning (preserving events whose projected 
fields are absent as NULL), or conservatively disable the merged-group 
optimizations for them. Add projection and predicate regressions alongside the 
two existing unfiltered window tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to