airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4068248446
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -79,6 +80,16 @@ Token* PinyinFilter::next(Token* token) {
current_token_text_ = std::string(token->termBuffer<char>(),
token->termLength<char>());
current_start_offset_ = token->startOffset();
current_end_offset_ = token->endOffset();
+ if (!config_->ignorePinyinOffset) {
+ auto source_byte_offsets = get_source_byte_offsets();
Review Comment:
Fixed in 73cd9c7fc95. Reproduced first: `keyword -> pinyin(full pinyin) ->
pinyin(letters, ignore_pinyin_offset=false)` on `刘德华` mapped `l`, `i`, `u` to
`[0,3)`, `[3,6)`, `[6,9)` because the second filter read the keyword
tokenizer's rune map through the first filter's default delegation.
`PinyinFilter` now overrides the three provenance getters: an unchanged
original candidate publishes exact rune boundaries (derived from the current
runes, so trimming stays consistent), every other candidate publishes a
conservative span covering its own source range, and `next()` reads its
upstream through the `DorisTokenFilter::` base implementation. Chained letters
now map to their Chinese rune (`[0,3)` for `l`/`i`/`u`, `[3,6)` for `d`/`e`,
...), and `TestChainedPinyinFiltersPublishCandidateSpansAndReset` covers the
chain, reset/reuse, and the exact passthrough of `keep_original` into a
following Pinyin filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
