jayzhan211 opened a new pull request, #24587: URL: https://github.com/apache/datafusion/pull/24587
## Rationale for this change Hash joins on a dictionary-encoded integer join key misbehave when dynamic filter pushdown is active. A dictionary array can carry NULL in its *values* while its key bitmap has no physical nulls (`null_count() == 0` but `logical_null_count() > 0`), and two places in the hash join only looked at the physical count: 1. **Wrong results:** for a `NullEqualsNull` join, `keys_have_null` was computed with `null_count()`, so the pushed dynamic filter was not widened with `key IS NULL` and pruned the probe-side NULL row that should have null-matched the build-side NULL. 2. **Query error:** `try_create_array_map` inferred "the key type is supported by `ArrayMap`" from the presence of min/max bounds. But bounds are also collected for dynamic filters on *any* key type, so with a dictionary key and a filter consumer the build side failed with `Unsupported type for ArrayMap: Dictionary(Int32, Int32)` instead of falling back to the regular hash map. ## What changes are included in this PR? - `stream.rs`: compute `keys_have_null` with `logical_null_count()`. - `exec.rs`: gate `ArrayMap` (perfect hash join) creation on the key type via `is_perfect_hash_join_candidate` rather than on the presence of bounds; also use `logical_null_count()` for the `NullEqualsNull` check in the same function. - Rename `should_collect_min_max_for_perfect_hash` → `is_perfect_hash_join_candidate` with a doc comment, since it describes what is checked rather than a side effect. ## Are these changes tested? Yes. A new test, `test_null_equal_dynamic_filter_keeps_probe_nulls_for_build_logical_null`, builds a `NullEqualsNull` inner join over dictionary keys with a dynamic filter consumer on the probe side and asserts both that the pushed filter contains the `IS NULL` disjunct and that the NULL–NULL row is produced. Before this PR it fails with the `ArrayMap` error; with only the `ArrayMap` gate it fails on the missing `IS NULL`. Existing hash join unit tests and the `joins`, `push_down_filter_parquet`, and `explain_analyze` sqllogictests pass. ## Are there any user-facing changes? No API changes. Joins on dictionary-encoded integer keys with dynamic filter pushdown now return correct results instead of erroring / dropping rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
