gabotechs opened a new issue, #25617: URL: https://github.com/apache/datafusion/issues/25617
## Describe the bug An anti join estimates almost complete customer coverage from customer-key range statistics in orders, despite a third of customers having no orders. Missing NDVs are replaced by range bounds and treated as evidence of matching values. ## To Reproduce From the repository root, with the CLI fix from [PR #25570](https://github.com/apache/datafusion/pull/25570) applied: ```sh cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed repro_dir=$(mktemp -d) tpchgen-cli --scale-factor 1 --format parquet \ --parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data" cat > "$repro_dir/repro.sql" <<'SQL' SET datafusion.execution.target_partitions = 1; SET datafusion.optimizer.enable_dynamic_filter_pushdown = false; SELECT c_custkey FROM customer WHERE NOT EXISTS (SELECT 1 FROM orders WHERE o_custkey = c_custkey); SQL target/ci/dfbench statistics \ --path "$repro_dir/data" --query_path "$repro_dir/repro.sql" ``` Observed with `tpchgen-cli` 1.1.1 at [6c320561b5](https://github.com/apache/datafusion/commit/6c320561b5b1aef7a235a12435c3c96b62956c67). Inspect the SELECT reports; ignore the empty `SET` reports. | Operator | Node | Estimated rows | Actual rows | | ------------------- | ---- | -------------: | ----------: | | HashJoinExec (anti) | `0` | 1 | 50,004 | ## Expected behavior Distinguish NDV/range upper bounds from evidence of overlapping values. Use measured distinct counts, constraints or an explicit uncertain-overlap fallback. ## Additional context Both scan row counts are exact: 150,000 customers and 1,500,000 orders. This removes the extra predicates from TPC-H Q22. Part of #25610. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
