yihua opened a new pull request, #911:
URL: https://github.com/apache/hudi-rs/pull/911
## Description
closes #909
`file_group::pushdown::filter_pushdown` turns `ReadOptions.filters` into a
parquet `RowFilter` (the filter columns are decoded first, the rest only for
the rows that pass) and a row-group selector over the footer statistics, and
`FileGroupReader` hands them to the merge-on-read reader for every slice it
reads. The reader's existing gate decides what may run below a merge:
everything on a slice without log files, only record key filters when log files
merge, and nothing under a position-based merge. A file that stores a filter
column in a type other than the table's (one written before a widening, or with
a differing timezone hint) gets nothing pushed, since a literal compared in the
file's type can disagree with the same comparison in the table's; float columns
are left out of statistics pruning because NaN is absent from their ranges. The
row mask applied after the merge is unchanged, so what a filter means does not
move.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Unit tests in `pushdown.rs` check row-group selection and row filtering on
an in-memory file, the type gate, the float gate and a column the file lacks.
`test_filters_prune_row_groups_of_a_slice_read` shows a filtered slice read
fetching one row group of five. `table_read_filter_pushdown_tests` reads
copy-on-write and merge-on-read tables with pushed filters, including a log
record that loses to the base row under an ordering field, where a filter
pushed below the merge would have leaked the log value. The existing read
suites pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
