Shahzeb-DataEngineer opened a new pull request, #26198:
URL: https://github.com/apache/datafusion/pull/26198
## Which issue does this PR close?
- Part of #17498
## Rationale for this change
`SET datafusion.execution.join_integer_prefilter_min_pruning_ratio` accepted
any float. Invalid values such as `2`, `-0.1`, or `NaN` succeeded and only
failed later, when a join executed with the integer prefilter enabled, with
`join_integer_prefilter_min_pruning_ratio must be finite and in [0, 1]`.
That is the same class of bug as `explain.format` in #17498: the option
documents a fixed contract ("Must be finite and in [0, 1]"), but invalid values
were silently stored. Following the review guidance on #17498 to land these as
smaller, single-option PRs (like #24400, #24541, #25626).
## What changes are included in this PR?
- Add a `ConfigUnitInterval` newtype in `datafusion-common` (validated
`FromStr`, following the `ConfigFilterSelectivity` / `ConfigTimeZone` pattern)
and use it as the type of
`ExecutionOptions::join_integer_prefilter_min_pruning_ratio`.
- Invalid values are now rejected at SET time with a clear error (`value
must be finite and in [0, 1], got <value>`), leaving the previous value in
place.
- Remove the now-redundant join-time check in
`PrefilterState::try_from_config`, since the invariant is enforced at the type
level.
- Regenerate the affected line in `docs/source/user-guide/configs.md` (doc
comment now says "validated when the config is set").
## What is the testing strategy for this PR?
- New unit test
`config::tests::test_join_integer_prefilter_min_pruning_ratio_validation`
covering valid values (incl. the `0`/`1` boundaries), out-of-range, negative,
`NaN`, infinite, and unparseable inputs, value-preservation on rejection, and
`RESET` back to the default.
- New `set_variable.slt` sqllogictest cases mirroring the existing
`default_filter_selectivity` cases.
- Existing `joins::integer_prefilter` tests still pass (7/7).
- `cargo fmt --all -- --check` clean; `cargo clippy -p datafusion-common -p
datafusion-physical-plan --all-targets --all-features -- -D warnings` clean.
## Are there any user-facing changes?
Yes. `SET datafusion.execution.join_integer_prefilter_min_pruning_ratio =
<invalid>` now fails immediately instead of being stored and failing at join
execution time. No API changes.
---
cc @viirya for review (author of the integer prefilter) and to trigger CI
for a first-time contributor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
