Pastoray opened a new pull request, #27514:
URL: https://github.com/apache/datafusion/pull/27514

   ## Which issue does this PR close?
   
   - Closes #27513.
   
   ## Rationale for this change
   
   A TopK query (`ORDER BY v LIMIT n`) normally skips Parquet row groups that 
the current TopK threshold rules out (`row_groups_pruned_dynamic_filter`). That 
stops working when the file stores the sort column as `REQUIRED` but the table 
schema declares it nullable, which is common when a table is given an explicit 
all-nullable schema so files with and without a column read as one table. 
Results stay correct, but every row group in the file gets decoded.
   
   With the repro from the issue (1M rows, 16 row groups, `LIMIT 100`):
   
   |                          | row groups pruned | bytes scanned |
   | ------------------------ | ----------------- | ------------- |
   | table schema = file schema | 15              | 97 KB         |
   | nullable table schema, before | 0            | 1.5 MB        |
   | nullable table schema, after  | 15           | 97 KB         |
   
   ## What changes are included in this PR?
   
   When the table field is nullable and the file field is not, the schema 
adapter wraps the column in a same-type `CAST` to the nullable field. The TopK 
filter then reaches the statistics pruner as `CAST(v) IS NULL OR CAST(v) < t`. 
`build_is_null_column_expr` only accepted a bare column, so the `IS NULL` 
branch was treated as unhandled and became `true`, which makes the whole `OR` 
unprunable. With matching schemas, or an `OPTIONAL` file column, there is no 
cast, which is why those cases already worked.
   
   `build_is_null_column_expr` in `datafusion/pruning/src/pruning_predicate.rs` 
now looks through a cast that is lossless and preserves nulls, using the 
existing `CastExpr::is_bigger_cast`, and builds the null-count expression for 
the column underneath. A cast that can introduce nulls (for example `Utf8` to 
`Int64`) is still left unhandled, so pruning stays conservative for it.
   
   ## What is the testing strategy for this PR?
   
   Three unit tests are added next to the existing `IS NULL` pruning tests in 
`pruning_predicate.rs`:
   
   - `prune_is_null_through_nullability_only_cast`: `IS NULL` and `IS NOT NULL` 
over the nullability-only cast.
   - `prune_topk_filter_through_nullability_only_cast`: the TopK shape `v IS 
NULL OR v < t` from the issue.
   - `prune_is_null_through_null_introducing_cast_is_not_pruned`: a cast that 
can introduce nulls must not be pruned.
   
   The first two fail without the change (no containers are pruned) and pass 
with it. I also ran the repro from the issue against the change, and the 
nullable-schema case now prunes 15 row groups. Existing tests in 
`datafusion-pruning`, `datafusion-datasource-parquet` and the 
`parquet`/`physical_optimizer` tests in `core_integration` pass.
   
   ## Are there any user-facing changes?
   
   No API changes. Queries that were already correct now skip row groups they 
previously decoded.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to