airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4070122876
##########
be/src/storage/index/inverted/token_filter/pinyin_filter_factory.cpp:
##########
@@ -62,7 +62,10 @@ TokenFilterPtr PinyinFilterFactory::create(const
TokenStreamPtr& in) {
auto filter = std::make_shared<PinyinFilter>(in, config_);
filter->initialize();
+ if (!config_->ignorePinyinOffset) {
+ filter->set_source_byte_offsets_enabled(true);
Review Comment:
Fixed in 7f514a6e2ec. Reproduced all three symptoms first: `中 -> zhong`
followed by an offset-aware Pinyin filter gave the letters `[0,1)` … `[4,5)`,
`a-b` was compacted to `ab` with `[0,2)`, and `nfkc_cf -> pinyin tokenizer`
left U+FB01 -> `fi` at `[0,2)` because the tokenizer never corrected offsets
through the char filter and published no provenance.
`PinyinTokenizer` now keeps the source range of every letter in the pending
ASCII buffer (so a candidate ends at its last letter's real byte and
`none_chinese_pinyin_tokenize` sub-tokens map to their own letters, with
`fixed_pinyin_offset` unchanged), corrects every emitted start/end through
`correct_source_offset()`, and publishes provenance per candidate: an exact
rune map through the shared `set_source_byte_offsets()` only when the candidate
is the untouched source slice, otherwise a conservative span over the
candidate's own (corrected) range via `get_conservative_source_byte_span()`.
`TestPinyinTokenizerPublishesCandidateProvenanceAndCorrectsOffsets` covers the
transformed `zhong`, the compacted `a-b`, and the ICU ligature chain with
reset/reuse to full-width input.
##########
be/src/storage/index/inverted/token_filter/ascii_folding_filter.cpp:
##########
@@ -42,7 +64,15 @@ Token* ASCIIFoldingFilter::next(Token* t) {
continue;
}
if (c >= 0x0080) {
+ const int32_t input_runes =
count_utf8_runes(std::string_view(buffer, length));
fold_to_ascii(buffer, length);
+ _rune_count_changed =
Review Comment:
Fixed in 7f514a6e2ec. Reproduced first: `keyword -> asciifolding ->
pinyin(letters)` on raw `0xff C3 86` reported `a` at `[0,1)` and `e` at `[1,2)`
because `count_utf8_runes()` returned `-1` and the filter kept delegating the
upstream map that no longer described the folded output.
`ASCIIFoldingFilter::next()` now treats a malformed input count as a
provenance change as well, so the folded token publishes the delegated
conservative span (the whole upstream token, `[0,3)` here) instead of an exact
map. `TestAsciiFoldingMalformedInputPublishesConservativeSpan` covers the
malformed case with and without `preserve_original` plus a reset to valid input.
##########
be/src/storage/index/inverted/token_filter/word_delimiter_filter.h:
##########
@@ -35,6 +36,14 @@ class WordDelimiterFilter : public DorisTokenFilter {
Token* next(Token* t) override;
void reset() override;
+ std::span<const int32_t> get_source_byte_offsets() const override {
Review Comment:
Fixed in 7f514a6e2ec. Reproduced first: `keyword -> word_delimiter ->
pinyin(letters)` on raw `0xff 61` emitted `a` with `[0,2)` but no provenance,
and the following filter fell back to the term length and reported `[0,1)`, the
malformed byte.
`WordDelimiterFilter` now tracks whether the current token is a generated
part and overrides `get_conservative_source_byte_span()`: an exact slice still
wins, an upstream conservative span is passed through, and otherwise a
generated part claims the whole upstream token (`[0, saved_end -
saved_start)`), which is also the offset range it is published with.
Passthrough tokens keep delegating upstream.
`TestWordDelimiterWithoutUpstreamProvenancePublishesTokenSpan` covers the
leading malformed byte, an interior malformed byte (which is not a delimiter,
so the token passes through unchanged and keeps exact byte positions), and
reset/reuse on valid input with exact slices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
