github-actions[bot] commented on code in PR #69698:
URL: https://github.com/apache/doris/pull/69698#discussion_r4395744755
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -801,6 +854,27 @@ float CollectionStatistics::get_or_calculate_avg_dl(const
std::wstring& lucene_c
return avg_dl;
}
+std::vector<std::wstring> CollectionStatistics::get_prefix_expansions(
+ const std::wstring& lucene_col_name, const std::wstring& prefix) {
+ const auto field = _prefix_expansions.find(lucene_col_name);
+ if (field == _prefix_expansions.end()) {
+ return {};
+ }
+ const auto expansions = field->second.find(prefix);
+ if (expansions == field->second.end()) {
+ return {};
+ }
+ // A merge-on-write scan hides rows: a term whose documents are all hidden
was expanded in
+ // its segment but has no document frequency left, and its idf would be
that of an absent term.
+ std::vector<std::wstring> terms;
+ for (const auto& term : expansions->second.terms()) {
+ if (get_term_doc_freq_by_col(lucene_col_name, term) > 0) {
Review Comment:
[P2] Refill the scoring cap after excluding hidden-only terms. With
inverted_index_max_expansions=1, let segment A contain hidden `apple ra` plus
visible `apple zz`, and B contain visible `apple rb`. The bounded set retains
`{ra}`, then this filter removes it; `rb` was evicted even though B still
matches it. After compaction removes the hidden `ra`, the same visible rows
select `{rb}`, changing the matching row's score from 0.1823 to 0.8755 (N=2,
avgdl=2), so `score() > 0.5` returns different rows depending on
compaction/replica. Both CLucene and SNII use this path. Apply the scoring cap
consistently with visible term frequencies, refilling after hidden-only
candidates are excluded, and cover this before/after-compaction case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
