mazne commented on issue #27513:
URL: https://github.com/apache/datafusion/issues/27513#issuecomment-6970275326

   A narrower trigger, from more runs of the same repro (DataFusion 55.2.0, one 
file of 16 row groups, `pushdown_filters = true`, 4 target partitions, `ORDER 
BY time LIMIT 100`, file column `REQUIRED`, table schema nullable):
   
   | ORDER BY | `row_groups_pruned_dynamic_filter` | `bytes_scanned` |
   |---|---|---|
   | `time ASC NULLS FIRST` | **0** | 558 KB |
   | `time ASC NULLS LAST` | 15 | 43 KB |
   | `time DESC NULLS LAST` | 15 | 8 KB |
   | `time DESC NULLS FIRST` | **0** | 554 KB |
   
   With an `OPTIONAL` file column, or the table read with the file's own 
(required) schema, all four orders prune 15 of 16 groups.
   
   So pruning is lost only when nulls sort first. That is when the TopK 
threshold carries an `IS NULL OR` term (`time IS NULL OR time < x`). Our guess 
is that the per-file expression rewrite in the Parquet opener simplifies `time 
IS NULL` to `false` for the required file column. The result is a new 
expression that no longer tracks the dynamic filter, so later threshold updates 
never reach the row-group pruner. `num_predicate_creation_errors` stays 0, and 
the row filter still applies the threshold (the rows are decoded, then dropped).
   
   We are working around it by planning `NULLS LAST` when statistics show the 
sort column has no nulls.
   
   A possible fix: when the opener rewrites the predicate for a file's physical 
schema, keep a `DynamicFilterPhysicalExpr` as a dynamic node (rewrite its 
current expression on each update, rather than snapshotting and simplifying it 
once), so threshold updates keep reaching the row-group pruner. Alternatively, 
skip the `IS NULL` → `false` simplification inside dynamic filters. We haven't 
tried either upstream yet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to