SteNicholas opened a new issue, #1445:
URL: https://github.com/apache/paimon-rust/issues/1445

   ### Search before asking
   
   - [x] I searched existing issues and open pull requests. No issue or active 
PR covering this specific scope was found on November 18, 2026.
   
   ### Motivation
   
   Data Evolution append tables reconstruct rows from column files aligned by 
row ID and file version. Rust already prunes whole row-ID groups using manifest 
statistics and evaluates file indexes, but multi-source column merging opens 
each source [without data 
predicates](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L1996-L2004).
   
   A predicate may retain the whole group even when Parquet row-group 
statistics or page indexes in the current column provider could exclude most of 
its rows. Independently filtering each source would be incorrect because the 
sources must remain positionally aligned.
   
   Java's [candidateRanges / 
computeCandidateRanges](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java#L1184-L1319)
 provides a reference: compute candidate row-ID ranges using the files that 
supply the effective predicate columns, then apply the same selection to every 
source in the merged group.
   
   ### Solution
   
   Start with Data Evolution tables (`data-evolution.enabled=true`, row 
tracking, no primary key, `bucket=-1`) and groups consisting exclusively of 
ordinary Parquet column files sharing the same complete row-ID range.
   
   - Resolve field ownership by stable field IDs, historical schema and file 
version. Only the winning provider's statistics may prune a column; older 
overwritten values must never exclude a live row.
   - Translate supported predicates to Parquet row-group/page candidate ranges. 
A predicate must be evaluable entirely within its winning provider before using 
that provider's metadata; preserve safe AND/OR semantics and fall back 
conservatively for unsupported expressions or uncertain statistics.
   - Convert file-relative ranges to logical row IDs and combine them with 
caller row ranges, file-index selections and deletion-vector handling. Pass one 
consistent positional selection to every contributing source; retain exact 
residual evaluation after reconstruction.
   - Reuse metadata loaded during candidate planning when opening readers, 
avoiding a second footer/index read.
   - Align with Java's `data-evolution.merged-read.stats-pushdown.enabled` 
option, including a disabled path for result/performance comparison.
   - Follow Java's conservative fallback: inability to establish safe pruning 
reads more rows, not fewer. Lost/corrupt-file errors remain governed by the 
normal read path and configured policies.
   
   #### Acceptance criteria
   
   - [ ] Enabled and disabled paths return identical results for multiple 
partial-column versions, NULL values, schema renames and dropped/recreated 
names.
   - [ ] An obsolete file's statistics cannot prune rows whose current values 
match.
   - [ ] Different row-group/page boundaries across sources do not change row 
alignment.
   - [ ] Caller ranges, file-index selections, deletion vectors, projection, 
LIMIT and residual predicates compose correctly.
   - [ ] Missing indexes, unsupported predicates/types, nested-field layouts 
and dedicated BLOB/Vector groups use a safe fallback.
   - [ ] Counting-I/O tests and a reproducible selective-read benchmark show 
reduced data bytes/decoded rows, and verify that metadata is reused. Report 
nonselective overhead as well.
   - [ ] Existing whole-group pruning and ordinary Parquet scan tests remain 
covered.
   
   ### Anything else?
   
   Rust references:
   
   - [Existing whole-group statistics 
pruning](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/table_scan.rs#L3147-L3174).
   - [Existing file-index selection and merged-source 
setup](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L754-L825).
   - [Winning-field file-index 
evaluation](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L1081-L1153).
   
   Java option: 
[DATA_EVOLUTION_MERGED_READ_STATS_PUSHDOWN_ENABLED](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java#L2782-L2792).
   
   This is a group-level extension, distinct from #780's existing FileIndex 
integration and #1025's selected-column Parquet page-index loading. Coordinate 
with those paths and reuse their metadata facilities.
   
   Dedicated BLOB/Vector groups, nested-field merging, primary-key LSM reads 
and concurrent source prefetch are outside the first version.
   
   Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, 
compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These 
findings come from source inspection; no new runtime reproduction or 
performance measurement is claimed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to