mazne commented on issue #27513: URL: https://github.com/apache/datafusion/issues/27513#issuecomment-6970275326
A narrower trigger, from more runs of the same repro (DataFusion 55.2.0, one file of 16 row groups, `pushdown_filters = true`, 4 target partitions, `ORDER BY time LIMIT 100`, file column `REQUIRED`, table schema nullable): | ORDER BY | `row_groups_pruned_dynamic_filter` | `bytes_scanned` | |---|---|---| | `time ASC NULLS FIRST` | **0** | 558 KB | | `time ASC NULLS LAST` | 15 | 43 KB | | `time DESC NULLS LAST` | 15 | 8 KB | | `time DESC NULLS FIRST` | **0** | 554 KB | With an `OPTIONAL` file column, or the table read with the file's own (required) schema, all four orders prune 15 of 16 groups. So pruning is lost only when nulls sort first. That is when the TopK threshold carries an `IS NULL OR` term (`time IS NULL OR time < x`). Our guess is that the per-file expression rewrite in the Parquet opener simplifies `time IS NULL` to `false` for the required file column. The result is a new expression that no longer tracks the dynamic filter, so later threshold updates never reach the row-group pruner. `num_predicate_creation_errors` stays 0, and the row filter still applies the threshold (the rows are decoded, then dropped). We are working around it by planning `NULLS LAST` when statistics show the sort column has no nulls. A possible fix: when the opener rewrites the predicate for a file's physical schema, keep a `DynamicFilterPhysicalExpr` as a dynamic node (rewrite its current expression on each update, rather than snapshotting and simplifying it once), so threshold updates keep reaching the row-group pruner. Alternatively, skip the `IS NULL` → `false` simplification inside dynamic filters. We haven't tried either upstream yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
