github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4068157500
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -79,6 +80,16 @@ Token* PinyinFilter::next(Token* token) {
current_token_text_ = std::string(token->termBuffer<char>(),
token->termLength<char>());
current_start_offset_ = token->startOffset();
current_end_offset_ = token->endOffset();
+ if (!config_->ignorePinyinOffset) {
+ auto source_byte_offsets = get_source_byte_offsets();
Review Comment:
[P1] Publish output-aligned provenance for each PinyinFilter candidate
instead of continuing to delegate the pre-transform map. In a valid `keyword ->
pinyin -> pinyin(ignore_pinyin_offset=false)` chain, the first filter turns
`刘德华` into candidates such as ASCII `liu` but still exposes the keyword map
`[0,3,6,9]`. The second filter sees three ASCII runes and a four-boundary map,
so its cardinality check succeeds and it assigns `l`, `i`, and `u` the
unrelated Chinese spans `[0,3)`, `[3,6)`, and `[6,9)` (or lets whole `liu` end
at 9 instead of 3). Give every emitted candidate an exact or conservative span
and cover chained Pinyin filters plus reset/reuse.
##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -202,34 +254,35 @@ IndexPolicyMgr::build_analyzer_config_from_policy(const
TIndexPolicy& index_poli
}
const auto& tokenizer_name = tokenizer_it->second;
- // Use normalized name for case-insensitive lookup
std::string normalized_tokenizer_name = normalize_name(tokenizer_name);
- if (_name_to_id.contains(normalized_tokenizer_name)) {
- const auto& tokenizer_policy =
_policys[_name_to_id[normalized_tokenizer_name]];
- auto type_it = tokenizer_policy.properties.find(PROP_TYPE);
- if (type_it == tokenizer_policy.properties.end()) {
+ if (const auto* tokenizer_policy =
find_policy_by_name_locked(tokenizer_name);
Review Comment:
[P1] Reject an exact tokenizer binding whose policy family is not
`TOKENIZER`. Replay intentionally retains and pushes legacy case-distinct
policies, so a state with exact `Foo=CHAR_FILTER(type=empty)`, normalized
`foo=TOKENIZER(type=standard)`, and analyzer `tokenizer=Foo` reaches this
branch without current DDL validation. This code consumes `type=empty`; because
`empty` is also registered as a tokenizer factory, BE silently builds
`EmptyTokenizer` rather than selecting the legacy tokenizer or failing.
`process_filter_configs()` already enforces the expected family for both filter
kinds. Mirror that guard here and cover replay/push materialization with a
cross-family exact collision.
##########
be/src/storage/index/inverted/analyzer/ik/IKTokenizer.cpp:
##########
@@ -31,27 +90,83 @@ Token* IKTokenizer::next(Token* token) {
return nullptr;
}
- std::string& token_text = tokens_text_[buffer_index_++];
+ TokenData& token_data = tokens_[buffer_index_++];
// full-width to half-width, and lowercase
// TODO(ryan19929): do regularizeString in fillBuffer.
- CharacterUtil::regularizeString(token_text, this->lowercase);
- size_t size = std::min(token_text.size(),
static_cast<size_t>(LUCENE_MAX_WORD_LEN));
- token->setNoCopy(token_text.data(), 0, static_cast<int32_t>(size));
+ if (source_byte_offsets_enabled_) {
+ current_source_byte_offsets_ =
+ regularize_with_source_byte_offsets(token_data.text,
this->lowercase);
+ } else {
+ CharacterUtil::regularizeString(token_data.text, this->lowercase);
+ current_source_byte_offsets_.clear();
+ }
+ current_token_ = &token_data;
+ const int32_t corrected_start =
+ source_char_filter_ == nullptr
+ ? token_data.start_offset
+ :
source_char_filter_->correct_offset(token_data.start_offset);
+ if (source_char_filter_ != nullptr && source_byte_offsets_enabled_) {
Review Comment:
[P1] Apply the generic repeated-boundary conversion to IK's
character-filter-corrected provenance. With `nfkc_cf`, source U+FB01 becomes
`fi`, so this loop converts IK boundaries `[0,1,2]` to `[0,3,3]`; IK publishes
only that start vector and no end intervals. A directly following offset-aware
Pinyin filter with `keep_none_chinese_together=false` treats those as the two
ASCII rune intervals and emits `i` at `[3,3)`.
`DorisTokenizer::set_source_byte_offsets()` already converts this pattern to
conservative start/end spans, but IK bypasses it. Reuse that invariant and
cover the ICU-ligature -> IK -> Pinyin chain with reset/reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
