airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4068247555
##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -202,34 +254,35 @@ IndexPolicyMgr::build_analyzer_config_from_policy(const
TIndexPolicy& index_poli
}
const auto& tokenizer_name = tokenizer_it->second;
- // Use normalized name for case-insensitive lookup
std::string normalized_tokenizer_name = normalize_name(tokenizer_name);
- if (_name_to_id.contains(normalized_tokenizer_name)) {
- const auto& tokenizer_policy =
_policys[_name_to_id[normalized_tokenizer_name]];
- auto type_it = tokenizer_policy.properties.find(PROP_TYPE);
- if (type_it == tokenizer_policy.properties.end()) {
+ if (const auto* tokenizer_policy =
find_policy_by_name_locked(tokenizer_name);
Review Comment:
Fixed in 4bb33c575b3. Reproduced first: with an exact `AnalyzerTokenizer`
CHAR_FILTER policy (`type=empty`), a normalized `analyzertokenizer` TOKENIZER
policy and an analyzer with `tokenizer=AnalyzerTokenizer`, `get_policy_by_name`
built the analyzer without any error
(`ReplayedPoliciesRejectWrongExactTokenizerType` failed on the missing
exception).
`build_analyzer_config_from_policy` now rejects an exact tokenizer binding
whose policy family is not `TOKENIZER` with the same "has type X but expected
TOKENIZER" message that `process_filter_configs` already uses for both filter
kinds. The new test replays the cross-family exact collision and asserts the
mismatch error.
##########
be/src/storage/index/inverted/analyzer/ik/IKTokenizer.cpp:
##########
@@ -31,27 +90,83 @@ Token* IKTokenizer::next(Token* token) {
return nullptr;
}
- std::string& token_text = tokens_text_[buffer_index_++];
+ TokenData& token_data = tokens_[buffer_index_++];
// full-width to half-width, and lowercase
// TODO(ryan19929): do regularizeString in fillBuffer.
- CharacterUtil::regularizeString(token_text, this->lowercase);
- size_t size = std::min(token_text.size(),
static_cast<size_t>(LUCENE_MAX_WORD_LEN));
- token->setNoCopy(token_text.data(), 0, static_cast<int32_t>(size));
+ if (source_byte_offsets_enabled_) {
+ current_source_byte_offsets_ =
+ regularize_with_source_byte_offsets(token_data.text,
this->lowercase);
+ } else {
+ CharacterUtil::regularizeString(token_data.text, this->lowercase);
+ current_source_byte_offsets_.clear();
+ }
+ current_token_ = &token_data;
+ const int32_t corrected_start =
+ source_char_filter_ == nullptr
+ ? token_data.start_offset
+ :
source_char_filter_->correct_offset(token_data.start_offset);
+ if (source_char_filter_ != nullptr && source_byte_offsets_enabled_) {
Review Comment:
Fixed in 73cd9c7fc95. Reproduced first: `nfkc_cf` char filter -> IK ->
Pinyin (`keep_none_chinese_together=false`, `ignore_pinyin_offset=false`) on
U+FB01 emitted `f` at `[0,3)` and `i` at `[3,3)`, exactly the zero-width span
described here.
The repeated-boundary conversion of
`DorisTokenizer::set_source_byte_offsets()` moved into a shared
`publish_source_byte_offsets()` helper, and `IKTokenizer::next()` now publishes
its char-filter-corrected boundaries through it (also enabling the base
tracking flag and clearing the base vectors on reset), so both letters keep the
whole ligature span `[0,3)`.
`TestIKExpandedLigatureOffsetsStayConservativeAndReset` covers the ligature
chain plus a reset to full-width input with exact per-letter spans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
