github-actions[bot] commented on code in PR #70262:
URL: https://github.com/apache/doris/pull/70262#discussion_r4504931550
##########
be/src/storage/index/inverted/clucene/inverted_index_writer.cpp:
##########
@@ -200,7 +197,8 @@ Result<std::shared_ptr<lucene::analysis::Analyzer>>
InvertedIndexColumnWriter<field_type>::create_analyzer(
const InvertedIndexAnalyzerConfig& analyzer_config) {
try {
- return
inverted_index::InvertedIndexAnalyzer::create_analyzer(&analyzer_config);
+ return std::make_shared<inverted_index::LuceneAnalyzerBridge>(
Review Comment:
[P2] Preserve tokenizer reuse for V1/V2 scalar writes
For a scalar STRING index using `parser=icu` or a custom `word_delimiter`
analyzer, `new_char_token_stream()` still calls `reusableTokenStream()` on
every row. With this factory change, only the outer bridge is reused:
`LuceneBackedTokenStream::reset()` destroys its old stream and calls the
underlying analyzer's `tokenStream()`, rebuilding its components for every
value. Previously `ICUAnalyzer`/`CustomAnalyzer::reusableTokenStream()` built
them once per writer. This now clones ICU iterators per row and copies the
entire configured `protected_words` set per row for `word_delimiter`, adding
dictionary-sized work even for short values during loads/index builds. Preserve
writer- or stream-local component reuse beneath the bridge; switching the
shared query analyzer globally to mutable CLucene reuse would break concurrent
stream isolation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
