puranjay2597 opened a new pull request, #67436:
URL: https://github.com/apache/doris/pull/67436
Supersedes #60799 (GitHub won't allow reopening a PR whose branch was
force-pushed after it was closed).
## What problem does this PR solve?
Adds 3 built-in scalar functions for fuzzy string matching and similarity
scoring, useful for record deduplication, search ranking, and data quality
workflows:
| Function | Return type | Description |
|---|---|---|
| `jaro(str1, str2)` | DOUBLE | Jaro similarity `[0.0, 1.0]` |
| `jaro_winkler(str1, str2)` | DOUBLE | Jaro-Winkler similarity `[0.0, 1.0]`
(boosts strings sharing a common prefix) |
| `jaccard_similarity(str1, str2)` | DOUBLE | Jaccard similarity `[0.0,
1.0]` over the sets of distinct characters of the two strings |
All functions accept `VARCHAR`/`STRING` inputs, propagate NULL, and support
constant folding.
### Rebased and reworked since #60799
#60799 originally also proposed `levenshtein` and `damerau_levenshtein`.
Both now already exist on master under different names (`levenshtein` via
#60412, `damerau_levenshtein_distance` via #65278), so they've been **dropped
from this PR** to avoid duplicating functionality — only the 3 functions above
remain.
The remaining functions have been reworked from the original submission to
address review feedback on #60799:
- **UTF-8 support** (requested by @linrrzqqq): all 3 functions now correctly
handle multi-byte UTF-8 input instead of operating byte-by-byte, e.g.
`jaro_winkler('你好世界', '你好世间')` now compares by character. Implemented with an
ASCII fast path plus a UTF-8-aware path, following the exact pattern
established by `levenshtein`/`damerau_levenshtein_distance`
(`VStringFunctions::get_utf8_char_offsets` / `utf8_char_equal`).
- **Reduced duplication** (requested by @linrrzqqq): added a standalone
`jaro` function containing the core Jaro algorithm; `jaro_winkler` now calls it
directly instead of duplicating the matching/transposition logic.
- **Jaccard algorithm clarified** (requested by @linrrzqqq, who couldn't
find the byte-bigram approach in Wikipedia or ClickHouse): `jaccard_similarity`
is now a character-set Jaccard index — `|A ∩ B| / |A ∪ B|` over the sets of
distinct bytes (ASCII) or Unicode characters (UTF-8) — matching ClickHouse's
`stringJaccardIndex` (`FunctionsStringDistance.cpp`) instead of an unexplained
bigram scheme. Uses a `std::bitset<256>` for the ASCII path per the reviewer's
suggestion, and a hash set of UTF-8 characters otherwise.
- **FE constant folding**: added `jaro`/`jaro_winkler`/`jaccard_similarity`
fold-constant implementations in `StringArithmetic.java`, matching the
convention used by
`levenshtein`/`damerau_levenshtein_distance`/`hamming_distance`.
- **Input length guard**: all 3 functions reject inputs over 65535 bytes
with a clear error, guarding the O(m×n) Jaro matching window and the set
construction in `jaccard_similarity` against unbounded `STRING` inputs.
- **Expanded tests**: BE unit tests (`function_string_test.cpp`) and
regression tests now cover column-vs-column, column-vs-constant (both
directions), nullable columns, UTF-8, and over-length-input error cases, in
addition to the constant-only cases.
### Release note
Add 3 built-in string similarity functions: `jaro`, `jaro_winkler`,
`jaccard_similarity`.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] Yes. Will follow up with a doris-website PR, as done for #60412.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
