Ma77Ball opened a new pull request, #59859:
URL: https://github.com/apache/spark/pull/59859
### What changes were proposed in this pull request?
This PR fixes `Levenshtein.eval` (the interpreted code path of the
`levenshtein(str1, str2[, threshold])` expression) to return `null` when the
optional `threshold` argument evaluates to `null`. Previously the `Some(v)`
branch called `v.asInstanceOf[Int]` directly; when `v` is `null` Scala unboxes
it to `0` rather than throwing, so a null threshold was silently treated as a
threshold of `0`. The fix adds an `if (v == null) return null` guard next to
the existing null checks on the two string arguments. A regression test is
added to the "Levenshtein distance threshold" case in `StringExpressionsSuite`
using `checkEvaluation` (which evaluates both the interpreted and
generated-code paths): a null left string, a null right string, and a null
threshold each evaluate to `null`.
### Why are the changes needed?
`Levenshtein` is null-intolerant, so any null input must yield `null`. The
generated-code path (`genCodeWithThreshold`) already returns `null` for a null
threshold, but the interpreted path returned a non-null integer: for example
`levenshtein('kitten', 'sitting', threshold)` with a null `threshold` computed
`levenshteinDistance('kitten', 'sitting', 0)` and returned `-1`. The same query
therefore produced different results depending on whether whole-stage codegen
was active (`NULL` with codegen, `-1` without), which is a correctness bug. The
existing tests used only constant literal thresholds that are never a runtime
null, so they did not catch it.
### Does this PR introduce _any_ user-facing change?
Yes. `levenshtein(str1, str2, threshold)` now returns `NULL` when
`threshold` is `NULL`, on both the interpreted and codegen paths. Previously
the interpreted path returned a wrong non-null value (such as `-1`). The two
string arguments were already handled correctly and are unchanged.
### How was this patch tested?
Added null-input cases (null left, null right, null threshold) to the
"Levenshtein distance threshold" test in `StringExpressionsSuite`;
`checkEvaluation` runs both the interpreted and codegen paths and requires them
to agree. Ran `StringExpressionsSuite` (the "Levenshtein distance threshold"
and "Levenshtein distance" cases); both pass.
### Was this patch authored or co-authored using generative AI tooling?
Co-authored with Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
