github-actions[bot] commented on code in PR #68758:
URL: https://github.com/apache/doris/pull/68758#discussion_r4244238487
##########
be/src/storage/index/inverted/query_v2/boolean_query/operator_boolean_weight.h:
##########
@@ -149,127 +192,156 @@ class OperatorBooleanWeight : public Weight {
return std::dynamic_pointer_cast<DoNothingCombiner>(_score_combiner)
!= nullptr;
}
- struct EvalResult {
- roaring::Roaring true_bitmap;
- roaring::Roaring null_bitmap;
- };
-
- static EvalResult collect_eval_result(ScorerPtr scorer, const
NullBitmapResolver* resolver) {
- EvalResult result;
- if (!scorer) {
- return result;
- }
-
- uint32_t doc = scorer->doc();
- if (doc == TERMINATED) {
- doc = scorer->advance();
- }
- while (doc != TERMINATED) {
- result.true_bitmap.add(doc);
- doc = scorer->advance();
- }
-
- if (scorer->has_null_bitmap(resolver)) {
- const auto* bitmap = scorer->get_null_bitmap(resolver);
- if (bitmap != nullptr) {
- result.null_bitmap = *bitmap;
+ // Groups term clauses by source for batched reads, unscored intersections
or scored unions.
+ // Remaining clauses use their individual scorers.
+ std::vector<ListedTerms> listed_terms(const QueryExecutionContext&
context, bool scoring) {
+ const bool any_source = (_type == OperatorType::OP_AND && !scoring) ||
+ (_type == OperatorType::OP_OR && scoring);
Review Comment:
[P1] Keep the scored V3 OR path bounded for TopK. `any_source` now groups
term clauses even when the source does not batch reads; `scored_disjunction()`
visits every posting and retains all matching rows and scores before the scorer
reaches TopK. A broad two-term OR on a large V3 field therefore needs memory
proportional to segment matches even for `LIMIT 1`, whereas the previous
per-term buffered union streamed them. Restrict eager listing to sources that
require it, or provide a bounded scored union for V3.
##########
be/src/exprs/function/function_search.cpp:
##########
@@ -355,12 +402,8 @@ Status
FunctionSearch::evaluate_inverted_index_with_search_param_unguarded(
// so score() queries must execute scorers to populate
CollectionSimilarity.
const bool enable_scoring =
index_query_context != nullptr &&
index_query_context->collection_similarity != nullptr;
- // Also bypass the DSL cache when any referenced field is served by an
SNII reader.
auto* dsl_cache =
- enable_cache && !enable_scoring &&
-
!referenced_fields_contain_snii_reader(search_param.root, iterators)
- ? InvertedIndexQueryCache::instance()
- : nullptr;
+ enable_cache && !enable_scoring ?
InvertedIndexQueryCache::instance() : nullptr;
Review Comment:
[P1] Publish SNII SEARCH data and NULL results atomically. This change lets
SNII use the DSL cache, but a miss inserts the data bitmap before its separate
NULL entry. Another query can hit the data entry in between, treat the missing
NULL entry as empty, and make `NOT search('body:foo')` include a row where
nullable `body` is NULL. Store both bitmaps in one entry, or treat a missing
paired NULL entry as a miss and recompute both.
##########
be/src/exprs/function/function_search.cpp:
##########
@@ -355,12 +402,8 @@ Status
FunctionSearch::evaluate_inverted_index_with_search_param_unguarded(
// so score() queries must execute scorers to populate
CollectionSimilarity.
const bool enable_scoring =
index_query_context != nullptr &&
index_query_context->collection_similarity != nullptr;
- // Also bypass the DSL cache when any referenced field is served by an
SNII reader.
auto* dsl_cache =
- enable_cache && !enable_scoring &&
-
!referenced_fields_contain_snii_reader(search_param.root, iterators)
- ? InvertedIndexQueryCache::instance()
- : nullptr;
+ enable_cache && !enable_scoring ?
InvertedIndexQueryCache::instance() : nullptr;
Review Comment:
[P1] Include the expansion limit in SNII SEARCH cache identity. This change
admits SNII results to the DSL cache, but its key contains only the segment
prefix and TSearchParam. A PREFIX query such as `search('body:fo*')` first run
with `inverted_index_max_expansions=1` caches rows for only its first expanded
term; the same query on that segment with limit 2 reuses the bitmap and misses
rows from the second term. Include the effective limit in the key, or bypass
this cache for expansion clauses.
##########
build-support/check-header-deps.py:
##########
@@ -579,6 +608,18 @@ def load_includes():
return includes
+def hub_headers(hub, includes):
Review Comment:
[P2] Cover the new directory rule in the gate self-test. `hub_headers` now
expands directory entries for both layering rules and angle bans, but the
unchanged hygiene self-test injects forbidden includes only into legacy exact
headers. A regression in this new dispatch could leave the test green while the
gate silently misses violations. The build-support guide requires a red/green
injection for each gate behavior change; add one for a covered directory header
and its angle ban.
##########
be/src/storage/index/query/logical/search_lowering.cpp:
##########
@@ -0,0 +1,450 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/query/logical/search_lowering.h"
+
+#include <algorithm>
+#include <cctype>
+#include <charconv>
+#include <iterator>
+#include <string_view>
+#include <unordered_map>
+#include <utility>
+
+namespace doris::index_query::logical {
+
+using segment_v2::InvertedIndexQueryType;
+
+namespace {
+
+bool is_compound(const std::string& clause_type) {
+ return clause_type == "AND" || clause_type == "OR" || clause_type == "NOT"
||
+ clause_type == "OCCUR_BOOLEAN";
+}
+
+bool is_pattern_clause(const std::string& clause_type) {
+ return clause_type == "WILDCARD" || clause_type == "REGEXP" || clause_type
== "PREFIX";
+}
+
+// Clause types whose value is analyzed on a tokenized index.
+bool is_analyzed_clause(const std::string& clause_type) {
+ return clause_type == "TERM" || clause_type == "PHRASE" || clause_type ==
"MATCH" ||
+ clause_type == "ANY" || clause_type == "ALL";
+}
+
+Occur to_occur(TSearchOccur::type occur) {
+ switch (occur) {
+ case TSearchOccur::SHOULD:
+ return Occur::kShould;
+ case TSearchOccur::MUST_NOT:
+ return Occur::kMustNot;
+ case TSearchOccur::MUST:
+ default:
+ return Occur::kMust;
+ }
+}
+
+void append_terms(Token token, std::vector<std::string>* terms) {
+ if (token.is_single_term()) {
+ terms->push_back(std::move(std::get<std::string>(token.term)));
+ } else {
+ auto& many = std::get<std::vector<std::string>>(token.term);
+ terms->insert(terms->end(), std::make_move_iterator(many.begin()),
+ std::make_move_iterator(many.end()));
+ }
+}
+
+// Groups tokens that share a position into one slot, in place.
+std::vector<Token> group_by_position(std::vector<Token> tokens) {
+ size_t slots = 0;
+ for (size_t i = 0; i < tokens.size(); ++i) {
+ if (slots > 0 && tokens[slots - 1].position == tokens[i].position) {
+ Token& slot = tokens[slots - 1];
+ if (slot.is_single_term()) {
+ slot.term = std::vector<std::string>
{std::move(std::get<std::string>(slot.term))};
+ }
+ append_terms(std::move(tokens[i]),
&std::get<std::vector<std::string>>(slot.term));
+ continue;
+ }
+ if (slots != i) {
+ tokens[slots] = std::move(tokens[i]);
+ }
+ ++slots;
+ }
+ tokens.resize(slots);
+ return tokens;
+}
+
+std::vector<std::string> flatten(std::vector<Token> tokens) {
+ std::vector<std::string> terms;
+ terms.reserve(tokens.size());
+ for (auto& token : tokens) {
+ append_terms(std::move(token), &terms);
+ }
+ return terms;
+}
+
+// Sets `out` to `phrase` over `tokens`, or to any of their terms when they
take one position.
+void lower_phrase(Phrase phrase, std::vector<Token> tokens, Node* out) {
+ phrase.slots = group_by_position(std::move(tokens));
+ if (phrase.slots.size() > 1) {
+ out->value = std::move(phrase);
+ return;
+ }
+ out->value =
+ TermSet {.field = std::move(phrase.field), .terms =
flatten(std::move(phrase.slots))};
+}
+
+// Sets `out` to a phrase whose last slot is a prefix; a single token is just
that prefix.
+void lower_phrase_prefix(std::vector<Token> tokens, Node* out) {
+ std::vector<Token> slots = group_by_position(std::move(tokens));
+ if (slots.size() == 1 && slots.front().is_single_term()) {
+ out->value = Expand {.field = {},
+ .kind = ExpandKind::kPrefix,
+ .pattern =
std::move(std::get<std::string>(slots.front().term))};
+ return;
+ }
+ out->value = Phrase {.field = {}, .slots = std::move(slots), .prefix =
true};
+}
+
+// Sets `out` to a phrase whose first slot takes the terms that end with its
token and whose last
+// slot the terms that start with its token; a single token takes every term
that contains it.
+void lower_phrase_edge(std::vector<Token> tokens, Node* out) {
+ std::vector<Token> slots = group_by_position(std::move(tokens));
+ if (slots.size() == 1 && slots.front().is_single_term()) {
+ out->value = Expand {.field = {},
+ .kind = ExpandKind::kContains,
+ .pattern =
std::move(std::get<std::string>(slots.front().term))};
+ return;
+ }
+ out->value = Phrase {.field = {}, .slots = std::move(slots), .prefix =
true, .suffix = true};
+}
+
+// Removes a trailing " ~N" or " ~N+" and returns its slop and ordering rule.
+std::pair<int32_t, bool> take_slop(std::string_view* value) {
+ const size_t space = value->find_last_of(' ');
+ if (space == std::string_view::npos || value->substr(space + 1, 1) != "~")
{
+ return {};
+ }
+ std::string_view digits = value->substr(space + 2);
+ const bool ordered = digits.size() > 1 && digits.back() == '+';
+ if (ordered) {
+ digits.remove_suffix(1);
+ }
+ int32_t slop = 0;
+ if (digits.empty() ||
+ !std::ranges::all_of(digits, [](unsigned char c) { return
std::isdigit(c) != 0; }) ||
+ std::from_chars(digits.data(), digits.data() + digits.size(), slop).ec
!= std::errc()) {
+ return {};
+ }
+ *value = value->substr(0, space);
+ return {slop, ordered};
+}
+
+NodePtr lower_direct_index_leaf(const std::string& clause_type, FieldRef field,
+ const std::string& value) {
+ if (clause_type == "TERM" || clause_type == "EXACT") {
+ return make_node(
+ Compare {.field = std::move(field), .op = CompareOp::kEqual,
.value = value});
+ }
+ return make_node(Unknown {.field = std::move(field)});
+}
+
+// WILDCARD, REGEXP and PREFIX match dictionary terms against a pattern. As in
Elasticsearch's
+// query_string, a prefix or a glob is normalized the way the index normalizes
its terms and is
+// never analyzed, and a regular expression is taken as written.
+Status lower_pattern_leaf(const std::string& clause_type, const FieldProps&
props,
+ FieldCatalog& catalog, FieldRef field, const
std::string& value,
+ NodePtr* out) {
+ if (clause_type == "REGEXP") {
+ // SEARCH matches a regular expression against whole terms on every
format.
+ *out = make_node(Expand {.field = std::move(field),
+ .kind = ExpandKind::kRegexp,
+ .pattern = "^(" + value + ")$"});
+ return Status::OK();
+ }
+ if (clause_type == "WILDCARD" && value == "*") {
+ *out = make_node(Exists {.field = std::move(field)});
+ return Status::OK();
+ }
+ const bool prefix = clause_type == "PREFIX";
+ // The DSL keeps a PREFIX value's trailing '*'.
+ std::string pattern =
+ prefix && value.ends_with('*') ? value.substr(0, value.size() - 1)
: value;
+ if (props.analyzed) {
+ std::string normalized;
+ RETURN_IF_ERROR(catalog.normalize(props, pattern, &normalized));
Review Comment:
[P1] Preserve WILDCARD operators during normalization. This call sends the
entire `a*b` pattern through a named policy normalizer. A valid `char_replace`
policy that replaces `*` with `x` turns it into literal `axb`, so
`search('body:a*b')` misses indexed term `acb`; both previous format paths kept
the `*` and matched it. Normalize only the literal spans, then restore the `*`
and `?` operators with their escape semantics.
##########
be/src/storage/index/inverted/query_v2/complete_null_bitmap.cpp:
##########
@@ -0,0 +1,367 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/inverted/query_v2/complete_null_bitmap.h"
+
+#include <algorithm>
+#include <array>
+#include <memory>
+#include <optional>
+#include <span>
+#include <utility>
+
+#include "core/custom_allocator.h"
+#include "storage/index/collection_similarity.h"
+#include "storage/index/inverted/query_v2/phrase_query/phrase_scorer.h"
+#include "storage/index/inverted/query_v2/scored_rows_scorer.h"
+#include "storage/index/inverted/query_v2/term_query/term_scorer.h"
+#include "storage/index/query/boolean/truth_set.h"
+#include "storage/index/query/exec/collect_postings.h"
+#include "storage/index/query/roaring_docid_sink.h"
+
+namespace doris::segment_v2::inverted_index::query_v2 {
+namespace {
+
+template <typename Next, typename Visitor>
+roaring::Roaring consume_true_rows(uint32_t doc, Next&& next, Visitor&& visit)
{
+ roaring::Roaring result;
+ std::array<uint32_t, 256> batch {};
+ size_t count = 0;
+ while (doc != TERMINATED) {
+ batch[count++] = doc;
+ visit(doc);
+ if (count == batch.size()) {
+ result.addMany(count, batch.data());
+ count = 0;
+ }
+ doc = next();
+ }
+ if (count != 0) {
+ result.addMany(count, batch.data());
+ }
+ return result;
+}
+
+template <typename ScorerType, typename Visitor>
+roaring::Roaring scan_candidate_rows(ScorerType& scorer, uint32_t doc,
+ const roaring::Roaring& candidates,
Visitor&& visit) {
+ const uint32_t last = candidates.maximum();
+ if (doc < candidates.minimum()) {
+ doc = scorer.seek(candidates.minimum());
+ }
+ const auto find_match = [&]() {
+ while (doc != TERMINATED && doc <= last && !candidates.contains(doc)) {
+ doc = scorer.advance();
+ }
+ return doc <= last ? doc : TERMINATED;
+ };
+ const auto first = find_match();
+ return consume_true_rows(
+ first,
+ [&]() {
+ if (doc == last) {
+ return TERMINATED;
+ }
+ doc = scorer.advance();
+ return find_match();
+ },
+ std::forward<Visitor>(visit));
+}
+
+template <typename ScorerType, typename Visitor>
+roaring::Roaring seek_candidate_rows(ScorerType& scorer, uint32_t doc,
+ const roaring::Roaring& candidates,
Visitor&& visit) {
+ auto candidate = candidates.begin();
+ const auto find_match = [&]() {
+ while (candidate.i.has_value && doc != TERMINATED) {
+ if (doc < *candidate) {
+ doc = scorer.seek(*candidate);
+ }
+ if (doc == *candidate) {
+ return doc;
+ }
+ candidate.equalorlarger(doc);
+ }
+ return TERMINATED;
+ };
+ const auto first = find_match();
+ return consume_true_rows(
+ first,
+ [&]() {
+ ++candidate;
+ return find_match();
+ },
+ std::forward<Visitor>(visit));
+}
+
+template <typename ScorerType, typename Visitor>
+roaring::Roaring consume_typed_true_rows(ScorerType& scorer, uint32_t doc,
+ const roaring::Roaring* candidates,
Visitor&& visit) {
+ const auto visit_doc = [&](uint32_t row) { visit(row, [&]() { return
scorer.score(); }); };
+ if (candidates == nullptr) {
+ return consume_true_rows(
+ doc, [&]() { return scorer.advance(); }, visit_doc);
+ }
+ if (candidates->isEmpty()) {
+ return {};
+ }
+ // Sequential reads avoid repeated cursor searches when candidates are
dense.
+ constexpr uint64_t seek_cost_multiplier = 4;
+ if (candidates->cardinality() * seek_cost_multiplier >= scorer.cost()) {
+ return scan_candidate_rows(scorer, doc, *candidates, visit_doc);
+ }
+ return seek_candidate_rows(scorer, doc, *candidates, visit_doc);
+}
+
+template <bool VisitRows, typename Visitor>
+roaring::Roaring consume_true_rows(Scorer& scorer, uint32_t doc, const
roaring::Roaring* candidates,
+ Visitor&& visit) {
+ if (auto* term = dynamic_cast<TermScorer*>(&scorer); term != nullptr) {
+ roaring::Roaring result;
+ index_query::RoaringDocIdSink sink(result);
+ THROW_IF_ERROR(index_query::collect_postings<VisitRows>(
+ term->postings_doc_set(), candidates, sink,
+ [&](uint32_t row, uint32_t frequency, uint32_t norm) {
+ visit(row, [&]() { return
term->score_for_posting(frequency, norm); });
+ }));
+ return result;
+ }
+ if (auto* phrase =
dynamic_cast<PhraseScorer<SegmentPostingsPtr>*>(&scorer);
+ phrase != nullptr) {
+ return consume_typed_true_rows(*phrase, doc, candidates,
std::forward<Visitor>(visit));
+ }
+ return consume_typed_true_rows(scorer, doc, candidates,
std::forward<Visitor>(visit));
+}
+
+struct ScoredRow {
+ uint32_t doc;
+ float score;
+};
+
+class MaterializedScorer final : public Scorer {
+public:
+ MaterializedScorer(const ScorerPtr& source, bool enable_scoring,
+ const NullBitmapResolver* resolver, const
roaring::Roaring* candidates)
+ : _iterator(_true_rows.begin()) {
+ if (const auto* nulls = source->get_null_bitmap(resolver); nulls !=
nullptr) {
+ _null_rows = candidates == nullptr ? *nulls : *nulls & *candidates;
+ }
+ std::optional<roaring::Roaring> true_candidates;
+ if (candidates != nullptr && !_null_rows.isEmpty()) {
+ true_candidates = *candidates - _null_rows;
+ candidates = &*true_candidates;
+ }
+ if (enable_scoring) {
+ _true_rows = consume_true_rows<true>(
+ *source, source->doc(), candidates, [&](uint32_t doc,
auto&& score) {
+ _scores.push_back({.doc = doc, .score = score()});
+ });
+ } else {
+ _true_rows = consume_true_rows<false>(*source, source->doc(),
candidates,
+ [](uint32_t, auto&&) {});
+ }
+ _iterator = _true_rows.begin();
+ }
+
+ uint32_t doc() const override {
+ if (!_scores.empty()) {
+ return _ordinal < _scores.size() ? _scores[_ordinal].doc :
TERMINATED;
+ }
+ return _iterator.i.has_value ? *_iterator : TERMINATED;
+ }
+ uint32_t size_hint() const override { return
static_cast<uint32_t>(_true_rows.cardinality()); }
+
+ uint32_t advance() override {
+ if (!_scores.empty()) {
+ if (_ordinal < _scores.size()) {
+ ++_ordinal;
+ }
+ } else if (_iterator.i.has_value) {
+ ++_iterator;
+ }
+ return doc();
+ }
+
+ uint32_t seek(uint32_t target) override {
+ const uint32_t current = doc();
+ if (target <= current) {
+ return current;
+ }
+ if (!_scores.empty()) {
+ // Unique sorted document IDs bound the skipped row count by the
document gap.
+ const size_t limit = std::min(_scores.size(), _ordinal + (target -
current));
+ const auto next = std::lower_bound(
+ _scores.begin() + _ordinal, _scores.begin() + limit,
target,
+ [](const ScoredRow& row, uint32_t value) { return row.doc
< value; });
+ _ordinal = next - _scores.begin();
+ } else {
+ _iterator.equalorlarger(target);
+ }
+ return doc();
+ }
+
+ float score() override { return _scores.empty() ? 0.0F :
_scores[_ordinal].score; }
+ bool has_null_bitmap(const NullBitmapResolver* /*resolver*/ = nullptr)
override {
+ return !_null_rows.isEmpty();
+ }
+ const roaring::Roaring* get_null_bitmap(
+ const NullBitmapResolver* /*resolver*/ = nullptr) override {
+ return &_null_rows;
+ }
+ const roaring::Roaring* get_true_bitmap() const override { return
&_true_rows; }
+
+private:
+ roaring::Roaring _true_rows;
+ roaring::Roaring _null_rows;
+ roaring::Roaring::const_iterator _iterator;
+ DorisVector<ScoredRow> _scores;
+ size_t _ordinal = 0;
+};
+
+// Equal UNKNOWN sets stay UNKNOWN under AND and OR.
+std::optional<roaring::Roaring> shared_null_bitmap(std::span<const ScorerPtr>
children,
+ const NullBitmapResolver*
resolver) {
+ const roaring::Roaring* shared = nullptr;
+ bool first = true;
+ for (const auto& child : children) {
+ const auto* nulls = child == nullptr ? nullptr :
child->get_null_bitmap(resolver);
+ if (nulls != nullptr && nulls->isEmpty()) {
+ nulls = nullptr;
+ }
+ if (first) {
+ shared = nulls;
+ first = false;
+ } else if (shared != nulls &&
+ (shared == nullptr || nulls == nullptr || *shared !=
*nulls)) {
+ return std::nullopt;
+ }
+ }
+ return shared == nullptr ? roaring::Roaring() : *shared;
+}
+
+} // namespace
+
+index_query::TruthSet collect_truth_set(const ScorerPtr& scorer, const
NullBitmapResolver* resolver,
+ const roaring::Roaring* candidates) {
+ index_query::TruthSet result;
+ if (scorer == nullptr) {
+ return result;
+ }
+ // Candidates dense enough to scan cost less to intersect once than to
test row by row.
+ constexpr uint64_t dense_candidate_factor = 4;
+ if (candidates != nullptr &&
+ candidates->cardinality() * dense_candidate_factor >= scorer->cost()) {
+ result = collect_truth_set(scorer, resolver);
+ result.true_rows &= *candidates;
+ result.null_rows &= *candidates;
+ return result;
+ }
+ if (const auto* nulls = scorer->get_null_bitmap(resolver); nulls !=
nullptr) {
+ result.null_rows = candidates == nullptr ? *nulls : *nulls &
*candidates;
+ }
+ if (const auto* truths = scorer->get_true_bitmap(); truths != nullptr) {
+ result.true_rows = candidates == nullptr ? *truths : *truths &
*candidates;
+ } else {
+ uint32_t doc = scorer->doc();
+ if (doc == TERMINATED) {
+ doc = scorer->advance();
+ }
+ std::optional<roaring::Roaring> true_candidates;
+ if (candidates != nullptr && !result.null_rows.isEmpty()) {
+ true_candidates = *candidates - result.null_rows;
+ candidates = &*true_candidates;
+ }
+ result.true_rows =
+ consume_true_rows<false>(*scorer, doc, candidates,
[](uint32_t, auto&&) {});
+ }
+ return result;
+}
+
+void collect_true_rows(const ScorerPtr& scorer, roaring::Roaring* rows) {
+ if (const auto* truths = scorer->get_true_bitmap(); truths != nullptr) {
+ *rows |= *truths;
+ return;
+ }
+ auto collected =
+ consume_true_rows<false>(*scorer, scorer->doc(), nullptr,
[](uint32_t, auto&&) {});
+ if (rows->isEmpty()) {
+ rows->swap(collected);
+ } else {
+ *rows |= collected;
+ }
+}
+
+roaring::Roaring collect_scored_rows(const ScorerPtr& scorer, uint32_t
doc_base,
+ CollectionSimilarity& similarity) {
+ if (const auto* listed = dynamic_cast<const
ScoredRowsScorer*>(scorer.get());
+ listed != nullptr) {
+ const std::span<const uint32_t> rows = listed->rows();
+ const std::span<const float> scores = listed->scores();
+ for (size_t i = 0; i < rows.size(); ++i) {
+ similarity.collect(rows[i] + doc_base, scores[i]);
+ }
+ roaring::Roaring result;
+ result.addMany(rows.size(), rows.data());
+ return result;
+ }
+ return consume_true_rows<true>(
+ *scorer, scorer->doc(), nullptr,
+ [&](uint32_t row, auto&& score) { similarity.collect(row +
doc_base, score()); });
+}
+
+ScorerPtr materialize_scorer(ScorerPtr source, bool enable_scoring,
+ const NullBitmapResolver* resolver,
+ const roaring::Roaring* candidates) {
+ if (source->get_true_bitmap() != nullptr) {
+ return source;
+ }
+ return std::make_shared<MaterializedScorer>(source, enable_scoring,
resolver, candidates);
+}
+
+roaring::Roaring complete_null_bitmap(std::vector<ScorerPtr>& children, bool
intersection,
+ bool enable_scoring, const
NullBitmapResolver* resolver) {
+ if (auto shared = shared_null_bitmap(children, resolver);
shared.has_value()) {
+ return std::move(*shared);
+ }
+ const bool nullable = std::ranges::any_of(children, [resolver](const
ScorerPtr& child) {
+ return child != nullptr && child->has_null_bitmap(resolver);
+ });
+ if (!nullable) {
+ return {};
+ }
+
+ index_query::TruthSet result;
+ bool first = true;
+ for (auto& child : children) {
+ index_query::TruthSet value;
+ if (child != nullptr) {
+ child = materialize_scorer(std::move(child), enable_scoring,
resolver);
Review Comment:
[P2] Defer null completion for scored TopK scorers. With nullable terms from
different fields, `shared_null_bitmap` cannot apply, so this path consumes
every child and stores every score while constructing the scorer.
`FunctionSearch` then builds the scorer again after TopK to get its null
bitmap, repeating the full scan even when only one result was requested.
Compute nulls once at the later extraction step, or share that result with TopK.
##########
be/src/storage/index/inverted/analyzer/analyzer.cpp:
##########
@@ -255,6 +256,56 @@ std::vector<TermInfo>
InvertedIndexAnalyzer::get_analyse_result(
return get_analyse_result(reader, analyzer.get());
}
+Status InvertedIndexAnalyzer::analyze(std::string_view value, const
InvertedIndexAnalyzerCtx* ctx,
+ const std::map<std::string,
std::string>& properties,
+ std::vector<TermInfo>* out) {
+ try {
+ if (ctx == nullptr) {
+ *out = get_analyse_result(value, properties);
+ } else if (!ctx->requires_analysis()) {
+ // Every string, the empty one included, is a term of an
untokenized index.
+ *out = {TermInfo {.term = std::string(value)}};
+ } else if (const auto analyzer = ctx->get_analyzer(); analyzer !=
nullptr) {
+ auto reader = create_reader(ctx->char_filter_map);
+ reader->init(value.data(), static_cast<int32_t>(value.size()),
true);
+ *out = get_analyse_result(reader, analyzer.get());
+ } else {
+ *out = get_analyse_result(value, properties);
+ }
+ } catch (const CLuceneError& e) {
+ return
Status::Error<ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>("analyzing '{}' failed:
{}",
+ value,
e.what());
+ } catch (const Exception& e) {
+ return
Status::Error<ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>("analyzing '{}' failed:
{}",
+ value,
e.what());
+ }
+ return Status::OK();
+}
+
+std::string InvertedIndexAnalyzer::normalize(const std::string& value,
+ const std::map<std::string,
std::string>& properties) {
+ const std::string analyzer_name =
get_analyzer_name_from_properties(properties);
+ if (analyzer_name.empty() || is_builtin_analyzer(analyzer_name)) {
Review Comment:
[P1] Apply the index's character filter when normalizing SEARCH PREFIX. With
an SNII index configured to replace `-` with `x`, the writer indexes `a-bcd` as
`axbcd`, but this built-in analyzer branch leaves the prefix from
`search('body:a-b*')` as `a-b`, so expansion misses the row. The previous SNII
PREFIX path analyzed the prefix with `analyzer_ctx->char_filter_map`. Normalize
through the same index-level filter before the prefix lookup, and cover this
case in the parity tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
