SteNicholas opened a new pull request, #1469:
URL: https://github.com/apache/paimon-rust/pull/1469

   ### Purpose
   
   Closes #1445.
   
   A selective filter on a Data Evolution table can make the reader decode an 
entire group of column files. Older files can hold overwritten values, and the 
current providers can have different Parquet page boundaries. This change 
identifies candidate row IDs from the files that supply the filtered columns 
and reads those positions from every contributing file.
   
   ### Brief change log
   
   - Resolve column ownership by stable field ID, historical schema and file 
version. Use a file's row-group statistics and page indexes for a conjunct only 
when it supplies every referenced field. Evaluate predicates spanning providers 
on the reconstructed rows.
   - Intersect candidate row IDs with requested ranges and file-index hits, 
exclude deleted rows, and apply the same selection to every source. Attach row 
IDs per batch and evaluate the full predicate before `LIMIT`.
   - Reuse the Parquet footers and indexes loaded during planning. Fall back to 
row-group statistics when page positions or byte ranges are invalid, and 
restore the original selection when file and manifest row counts disagree.
   - Enable pruning for ordinary Parquet files that share a complete row-ID 
range on row-tracked append tables with `bucket=-1`. Dedicated BLOB and vector 
groups, nested-field evolution and ROW sidecar reads use their existing read 
paths.
   
   ### Tests
   
   - **4,996 core unit tests passed** with 8 ignored by default. **45 
integration tests passed** across bitmap merges, ROW sidecars, vector 
compaction, system columns, partition restoration and Parquet page indexes. 
Coverage includes renamed and recreated columns, nulls, mismatched page 
boundaries and corrupt offsets. Counting-I/O tests verify one footer fetch per 
provider and reuse of a separately fetched page index.
   - Ran the selective and full-range benchmark over 262,144 logical rows on an 
Apple M4 Pro. The selective filter returned the same row in both modes. 
Pushdown reduced rows delivered to the merge from 524,288 to 128, column-data 
bytes from 2,342,012 to 756, and median scan time from 61.8 ms to 13.0 ms. The 
full-range filter returned all rows in both modes, with a 4.4% increase in 
median scan time.
   - Formatting and full-workspace Clippy passed on Rust 1.98.0 with 
`--all-targets --workspace --features vortex -- -D warnings`.
   
   ### API and Format
   
   `CoreOptions::data_evolution_merged_read_stats_pushdown_enabled()` reads the 
Java-compatible table option 
`data-evolution.merged-read.stats-pushdown.enabled`. It defaults to `true`. Set 
it to `false` to disable merged-read statistics pruning. The reader uses 
metadata already stored in existing Parquet files.
   
   ### Documentation
   
   `docs/src/getting-started.md` describes when pruning applies, how it falls 
back and how to disable it. `docs/src/benchmark.md` includes the reproduction 
command, byte counts and seven-run median scan times for both filters.
   
   🤖 Assisted with Codex.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to