zhuqi-lucas commented on code in PR #23696:
URL: https://github.com/apache/datafusion/pull/23696#discussion_r3758237598
##########
datafusion/core/tests/parquet/dynamic_row_group_pruning.rs:
##########
@@ -585,3 +585,76 @@ async fn
dynamic_rg_pruning_fires_for_multi_column_sort_leading_tied() {
output.description(),
);
}
+
+/// Per-RG `fully_matched` `RowFilter` skip optimization.
+///
+/// Stats prove that every row of a fully-matched row group satisfies the
+/// pushdown predicate, so the parquet decoder can skip the per-row
+/// `RowFilter` for that RG entirely. The stream rebuilds the decoder at
+/// the boundary with an empty `RowFilter` and toggles back to the real
+/// one at the next non-fully-matched RG.
+///
+/// Layout: 4 RGs of 3 values each. Predicate `v >= 3` makes RG 0 a
+/// straddler (some rows fail) but RGs 1..=3 fully matched (every value
+/// >= 3 by stats). RG 0 keeps the row filter, then the toggle flips to
+/// "no filter" when we enter the fully-matched run.
Review Comment:
Thanks @adriangb for the review, good suggestion! Updated the test to `v >=
3 AND v <= 10` in 801b099f6 — RG 3 now straddles the upper bound (11, 12 fail),
so it covers the full toggle lifecycle (filter ON → OFF across the
fully-matched run → back ON) and asserts 11/12 are filtered out by the
reinstalled RowFilter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
