airborne12 commented on code in PR #66861:
URL: https://github.com/apache/doris/pull/66861#discussion_r3801727603
##########
be/src/storage/index/snii/query/internal/wildcard_matcher.h:
##########
@@ -43,37 +45,76 @@ namespace doris::snii::query::internal {
template <class Alloc = std::allocator<uint8_t>>
class WildcardMatcher {
public:
- explicit WildcardMatcher(std::string_view pattern) : pattern_(pattern) {}
+ explicit WildcardMatcher(std::string_view pattern)
+ : pattern_(pattern), pattern_valid_(is_valid_utf8(pattern)) {}
bool operator()(std::string_view text) {
+ if (!pattern_valid_ || !is_valid_utf8(text)) {
Review Comment:
Confirmed independently with a real writer-to-reader reproduction. Before
the fix, a raw keyword term `61FF` produced `a* -> {valid-only}`, `** ->
{valid-only}`, and `* -> {valid-only}` at the SNII query API, omitting the
malformed term. Commit bd9a2b530ac keeps query patterns strict UTF-8, uses
code-point transitions for valid dictionary terms, and falls back to the legacy
byte transitions only for malformed stored terms. The same corpus now returns
the raw row for `a*`, `**`, and `*` while retaining the UTF-8 `?` behavior.
Added both BE writer/reader coverage and a real SNII regression case using
`CAST(UNHEX(61FF) AS STRING)`; 23 affected ASAN UTs and the
generated-then-normal regression suite pass. This is query-only: no writer or
on-disk format change, so existing SNII indexes are fixed by upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
