viirya opened a new pull request, #25517:
URL: https://github.com/apache/datafusion/pull/25517
## Which issue does this PR close?
N/A
## Rationale for this change
Projection statistics currently infer a CAST's source type from its minimum
or
maximum value. When both extrema are absent, even a lossless widening cast
such
as `Int32` to `Int64` is treated as unsafe.
This discards otherwise valid statistics, including exact null counts and
distinct counts, and can reduce the quality of optimizer estimates.
## What changes are included in this PR?
- Add `ProjectionExprs::project_statistics_with_input_schema`, which uses the
input schema to identify lossless casts when typed extrema are unavailable.
- Preserve the existing conservative behavior for casts that may fail,
discard
values, or change ordering.
- Use the schema-aware method in `ProjectionExec`, extended statistics
projection, and file scan statistics projection.
- Add helper-level and `ProjectionExec` regression tests covering an `Int32`
to
`Int64` cast with absent extrema but exact null and distinct counts.
The existing `project_statistics` API remains available, so this is an
additive, non-breaking API change.
## What is the testing strategy for this PR?
Added:
- `test_project_statistics_safe_cast_without_extrema`
- `test_projection_statistics_safe_cast_without_extrema`
Ablation testing confirmed that the old implementation changes
`null_count = Exact(3)` to `Absent`. Both the helper-level and production
`ProjectionExec` paths preserve the exact null and distinct counts after the
fix.
Also verified:
- Targeted regression tests
- `cargo fmt --all -- --check`
- `git diff --check`
The full test suite is covered by CI.
## Are there any user-facing changes?
Optimizer statistics are now preserved through known-lossless casts even when
minimum and maximum statistics are unavailable.
This PR adds a public method but does not remove or change the existing API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
