SteNicholas opened a new issue, #1445: URL: https://github.com/apache/paimon-rust/issues/1445
### Search before asking - [x] I searched existing issues and open pull requests. No issue or active PR covering this specific scope was found on November 18, 2026. ### Motivation Data Evolution append tables reconstruct rows from column files aligned by row ID and file version. Rust already prunes whole row-ID groups using manifest statistics and evaluates file indexes, but multi-source column merging opens each source [without data predicates](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L1996-L2004). A predicate may retain the whole group even when Parquet row-group statistics or page indexes in the current column provider could exclude most of its rows. Independently filtering each source would be incorrect because the sources must remain positionally aligned. Java's [candidateRanges / computeCandidateRanges](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java#L1184-L1319) provides a reference: compute candidate row-ID ranges using the files that supply the effective predicate columns, then apply the same selection to every source in the merged group. ### Solution Start with Data Evolution tables (`data-evolution.enabled=true`, row tracking, no primary key, `bucket=-1`) and groups consisting exclusively of ordinary Parquet column files sharing the same complete row-ID range. - Resolve field ownership by stable field IDs, historical schema and file version. Only the winning provider's statistics may prune a column; older overwritten values must never exclude a live row. - Translate supported predicates to Parquet row-group/page candidate ranges. A predicate must be evaluable entirely within its winning provider before using that provider's metadata; preserve safe AND/OR semantics and fall back conservatively for unsupported expressions or uncertain statistics. - Convert file-relative ranges to logical row IDs and combine them with caller row ranges, file-index selections and deletion-vector handling. Pass one consistent positional selection to every contributing source; retain exact residual evaluation after reconstruction. - Reuse metadata loaded during candidate planning when opening readers, avoiding a second footer/index read. - Align with Java's `data-evolution.merged-read.stats-pushdown.enabled` option, including a disabled path for result/performance comparison. - Follow Java's conservative fallback: inability to establish safe pruning reads more rows, not fewer. Lost/corrupt-file errors remain governed by the normal read path and configured policies. #### Acceptance criteria - [ ] Enabled and disabled paths return identical results for multiple partial-column versions, NULL values, schema renames and dropped/recreated names. - [ ] An obsolete file's statistics cannot prune rows whose current values match. - [ ] Different row-group/page boundaries across sources do not change row alignment. - [ ] Caller ranges, file-index selections, deletion vectors, projection, LIMIT and residual predicates compose correctly. - [ ] Missing indexes, unsupported predicates/types, nested-field layouts and dedicated BLOB/Vector groups use a safe fallback. - [ ] Counting-I/O tests and a reproducible selective-read benchmark show reduced data bytes/decoded rows, and verify that metadata is reused. Report nonselective overhead as well. - [ ] Existing whole-group pruning and ordinary Parquet scan tests remain covered. ### Anything else? Rust references: - [Existing whole-group statistics pruning](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/table_scan.rs#L3147-L3174). - [Existing file-index selection and merged-source setup](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L754-L825). - [Winning-field file-index evaluation](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/table/data_evolution_reader.rs#L1081-L1153). Java option: [DATA_EVOLUTION_MERGED_READ_STATS_PUSHDOWN_ENABLED](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java#L2782-L2792). This is a group-level extension, distinct from #780's existing FileIndex integration and #1025's selected-column Parquet page-index loading. Coordinate with those paths and reuse their metadata facilities. Dedicated BLOB/Vector groups, nested-field merging, primary-key LSM reads and concurrent source prefetch are outside the first version. Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These findings come from source inspection; no new runtime reproduction or performance measurement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
