Dandandan commented on code in PR #20413:
URL: https://github.com/apache/datafusion/pull/20413#discussion_r2817872904
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -459,27 +465,55 @@ impl FileOpener for ParquetOpener {
// `row_filter` for details.
//
---------------------------------------------------------------------
- // Filter pushdown: evaluate predicates during scan
- if let Some(predicate) =
pushdown_filters.then_some(predicate).flatten() {
- let row_filter = row_filter::build_row_filter(
- &predicate,
- &physical_file_schema,
- builder.metadata(),
- reorder_predicates,
- &file_metrics,
- );
+ // When a dynamic filter is present, defer expensive string-column
+ // predicates to post-decode batch filtering to avoid blocking
+ // dynamic filter convergence via predicate cache overhead.
+ //
+ // Stats pruning still uses the full predicate (unaffected).
+ let mut leftover_filter: Option<Arc<dyn PhysicalExpr>> = None;
- match row_filter {
- Ok(Some(filter)) => {
- builder = builder.with_row_filter(filter);
- }
- Ok(None) => {}
- Err(e) => {
- debug!(
- "Ignoring error building row filter for
'{predicate:?}': {e}"
- );
- }
+ if let Some(predicate) =
pushdown_filters.then_some(predicate).flatten() {
+ let conjuncts = split_conjunction(&predicate);
+ let has_dynamic = conjuncts.iter().any(|c|
is_dynamic_physical_expr(c));
+
+ let (row_filter_pred, leftover_pred) = if has_dynamic {
+ let (deferred, rf_conjuncts): (Vec<_>, Vec<_>) =
+ conjuncts.into_iter().partition(|c| {
+ !is_dynamic_physical_expr(c)
Review Comment:
See
https://github.com/apache/datafusion/issues/20324#issuecomment-3915572132
We can make it a bit more general / overall improvement by not pushing
`RowFilter` if it is the final column of the scan.
We can track the columns that are used for all of the filters before it and
skip adding the filter if it is the last column to be scanned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
