JingsongLi commented on code in PR #1422:
URL: https://github.com/apache/paimon-rust/pull/1422#discussion_r4601208603
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -551,7 +571,12 @@ impl DataEvolutionReader {
if remaining == Some(0) {
break 'splits;
}
- if is_raw_convertible(&files) {
+ // Change events are physical file changes. Files of a
streaming
+ // split that are not column slices of one row range are
+ // independent changes and are read as they are stored.
+ if is_raw_convertible(&files)
+ || (split.is_streaming() &&
!is_data_evolution_column_slice_group(&files))
Review Comment:
[P1] Apply physical-event semantics before scan pruning
Selecting raw events only here is too late: TableScan has already applied
merged-row statistics and read-field pruning. On this PR's own two inserts
(values 1..4) → compaction → value-only update to 50 fixture, the full
timestamp window returns 8 physical events. With `with_projection(&["id"])`,
its scan drops the update-only file and returns 4 rows; with `with_filter(value
= 1)`, latest-provider stats (min=max=50) drop the entire group and return zero
instead of the original `(1,1)` event. Both are public-API reproductions on
this exact head. Reading the original unfiltered/full-width plan through the
same projected or filtered reader returns 8 or 1 respectively.
Please classify these physical-event groups before planner statistics/field
pruning and use event-safe per-file pruning (preserving events whose projected
fields are absent as NULL), or conservatively disable the merged-group
optimizations for them. Add projection and predicate regressions alongside the
two existing unfiltered window tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]