airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4059975351
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -79,6 +80,11 @@ Token* PinyinFilter::next(Token* token) {
current_token_text_ = std::string(token->termBuffer<char>(),
token->termLength<char>());
current_start_offset_ = token->startOffset();
current_end_offset_ = token->endOffset();
+ auto source_byte_offsets = get_source_byte_offsets();
+ current_source_byte_offsets_.assign(source_byte_offsets.begin(),
source_byte_offsets.end());
Review Comment:
Fixed in `298168706baf432dc7fcfb91fd080e02cc1957bc`. Basic, CharGroup,
Empty, ICU, and NGram tokenizers now publish document-relative source
boundaries for offset-aware Pinyin. ICU builds an exact UTF-16-to-UTF-8
boundary map; transformations that change rune counts use a conservative
whole-source-span map. NGram byte offsets also preserve `UChar32` instead of
narrowing code points. Added a Basic reset and multi-token regression plus
offset-aware Pinyin coverage for every custom tokenizer. Validation: targeted
ASAN tests 2/2, broader tokenizer/filter ASAN suite 150/150, full BE build
passed, and clang-tidy passed for all files added by the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
