discord9 opened a new pull request, #23966: URL: https://github.com/apache/datafusion/pull/23966
## Which issue does this PR close? - Closes #23963. ## Rationale for this change The scalar regex fast path for Dictionary arrays evaluates each dictionary value once and gathers the resulting booleans with `BooleanArray::take_iter`. That gather preserves null keys but drops null validity from the evaluated dictionary values. A key referencing a null dictionary value therefore becomes valid `false`, and negated regex operators can incorrectly turn it into `true`. This is distinct from #23722, which updates the array-pattern `regex_match_dyn` path. This PR fixes null propagation in `regex_match_dyn_scalar`. ## What changes are included in this PR? - Replace the validity-dropping dictionary gather with Arrow `take`, preserving both key validity and evaluated child validity. - Add a physical-expression regression test for `RegexMatch`, `RegexIMatch`, `RegexNotMatch`, and `RegexNotIMatch`. The test covers a null key and a key referencing a null dictionary value, and compares the Dictionary result with the equivalent cast-to-`Utf8` evaluation. ## Are these changes tested? Yes. - Focused regression: 1 passed. - `datafusion-physical-expr`: 1,649 passed, including the new regression. - Remaining workspace: 7,928 passed, 0 failed, 7 ignored; sqllogictest completed 499/499 files. - `datafusion` library: 461 passed. - `datafusion` core integration: 1,057 passed; four local memory-validation runners exceeded their host RSS thresholds and were isolated as environment-sensitive, unrelated to this patch. - `datafusion-cli`: 106 passed. - Formatting, workspace clippy with warnings denied, TOML formatting, license headers, changed-file typo checks, and `git diff --check` passed. ## Are there any user-facing changes? Dictionary-encoded string inputs now preserve SQL null semantics for scalar regex operators. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
