github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4057030782
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -140,7 +199,7 @@ private void
validateReferencedComponentsUsableLocked(String analyzerName, Index
Review Comment:
[P1] Validate the expected type for every exact nested filter. This loop now
resolves token filters exact-first but only checks isInvalid(), so replayed
legacy names such as exact I=CHAR_FILTER(type=char_replace) and normalized
i=TOKEN_FILTER(type=lowercase), with analyzer token_filter=I, pass FE and then
BE sends char_replace to TokenFilterFactory and throws (or can apply the wrong
family when names overlap). The replay/image compatibility path intentionally
retains these policies, so CREATE-time validation is not enough; char_filter
references and normalizer nested lists are likewise not checked here. Reject
non-TOKEN_FILTER/non-CHAR_FILTER bindings in FE and BE and add replay/image
tests for both nested properties.
##########
be/src/storage/index/inverted/token_filter/pinyin_filter_factory.cpp:
##########
@@ -62,7 +62,10 @@ TokenFilterPtr PinyinFilterFactory::create(const
TokenStreamPtr& in) {
auto filter = std::make_shared<PinyinFilter>(in, config_);
filter->initialize();
+ if (!config_->ignorePinyinOffset) {
Review Comment:
[P1] Preserve source spans through text-changing token filters. This new
flag makes Pinyin consume the upstream per-rune map, but ICUNormalizerFilter
rewrites token text and forwards the old map and endpoints unchanged. With
keyword -> token_filter icu_normalizer -> pinyin(ignore_pinyin_offset=false),
input U+FB01 followed by Chinese text (source bytes 0..6) becomes fi plus the
Chinese text but the map stays [0,3,6], so Pinyin falls back to normalized
bytes and reports liu as [2,5) (whole-token candidates end at 5) instead of
[3,6)/[0,6); reset repeats it. Transform provenance for
expansions/contractions, or explicitly force unprojectable outputs to the
original token span, and add reset/reuse coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
