xudong963 opened a new pull request, #23788: URL: https://github.com/apache/datafusion/pull/23788
## Which issue does this PR close? - Closes #23787. ## Rationale for this change Join cardinality estimation multiplies two `usize` row counts before applying the join-key NDV divisor. A large Cartesian-product intermediate can therefore overflow and panic even when the normalized cardinality is representable. Saturating the product before division would avoid the panic but produce an inaccurate estimate. ## What changes are included in this PR? - Compute the Cartesian-product intermediate with `u128` before applying the NDV divisor. - Preserve exact estimates when the normalized result fits in `usize`. - Cap an unrepresentable final result at `usize::MAX` and mark it inexact, consistent with existing statistics arithmetic. - Add regression coverage for exact and inexact statistics and for final-result overflow. ## Are these changes tested? Yes. The regression test was verified to fail on `main` with `attempt to multiply with overflow` and pass with this change. The following checks pass: - `cargo test -p datafusion-physical-plan joins::utils::tests::` - `cargo clippy --all-targets --all-features -- -D warnings` - `./dev/rust_lint.sh` - The workspace extended-test command from `AGENTS.md`, including Avro, JSON, backtrace, recursive protection, and Parquet encryption features ## Are there any user-facing changes? Queries with very large join statistics no longer panic during cardinality estimation. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
