airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4062735820
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -214,29 +228,82 @@ bool PinyinFilter::readTerm(Token* token) {
return false;
}
-bool PinyinFilter::processCurrentToken() {
- processed_candidate_ = true;
+bool PinyinFilter::prepareCurrentSource(std::vector<UChar32>&
source_codepoints) {
+ size_t source_start = 0;
+ size_t source_end = current_token_text_.size();
+ if (config_->trimWhitespace) {
+ source_start = current_token_text_.find_first_not_of(" \t\n\r");
+ if (source_start == std::string::npos) {
+ return false;
+ }
+ source_end = current_token_text_.find_last_not_of(" \t\n\r") + 1;
+ }
+ current_source_ = current_token_text_.substr(source_start, source_end -
source_start);
- if (!has_current_token_) {
+ if (current_source_.empty()) {
return false;
}
- current_source_ = current_token_text_;
-
- // Apply trimming if configured
- if (config_->trimWhitespace) {
- current_source_ = trim(current_source_);
+ current_runes_ = convertToRunes(current_source_, source_codepoints);
Review Comment:
Fixed in 20c49dde. The default ignore-offset path no longer snapshots
upstream maps/spans or builds `current_runes_`; it uses codepoint-only
decoding, and original runes are decoded only when an exact source map is
present. `reset()` releases retained vector and string capacity. Added a 64 KiB
default-mode ASAN test that verifies rune capacity remains zero before and
after reset; the related BE suite passes 147/147.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
