discord9 opened a new issue, #23963: URL: https://github.com/apache/datafusion/issues/23963
### Describe the bug `regex_match_dyn_scalar` evaluates a dictionary's values once and gathers the resulting booleans with `BooleanArray::take_iter`. The gather preserves null dictionary keys, but it does not preserve nulls in the evaluated dictionary values. As a result, a valid key that references a null dictionary value becomes a valid `false`. Negated regex operators then turn that value into `true`, which violates SQL three-valued logic and differs from evaluating the same input after casting it to `Utf8`. ### To Reproduce Construct a dictionary array whose values contain both a string and a null, and whose keys contain both valid keys and a null key. Evaluate it against a scalar pattern with: - `RegexMatch` - `RegexIMatch` - `RegexNotMatch` - `RegexNotIMatch` Compare the result with evaluation after casting the dictionary array to `Utf8`. The current dictionary-specialized path loses validity for keys that reference the null dictionary value; the cast path retains it. ### Expected behavior Scalar regex evaluation on dictionary input should preserve both dictionary-key validity and dictionary-value validity. Its values and null bitmap should match the equivalent cast-to-`Utf8` evaluation for all four regex operators. ### Additional context The dictionary/scalar optimization was introduced in #12768. Current `main` and the `54.1.0` tag still use the validity-dropping `take_iter` gather. #23722 is related but distinct: it addresses dictionary input in the array-pattern `regex_match_dyn` path, while this report concerns null propagation in `regex_match_dyn_scalar`. A downstream reproduction and focused fix are available in GreptimeTeam/datafusion#22. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
