github-actions[bot] commented on code in PR #69790:
URL: https://github.com/apache/doris/pull/69790#discussion_r4409110677
##########
be/src/storage/index/index_reader_helper.cpp:
##########
@@ -24,12 +24,16 @@ namespace {
bool can_score_match(const TabletIndex* index_meta) {
const auto& properties = index_meta->properties();
- // parser=none can inherit support_phrase=true from FE defaults. MATCH
statistics skip
- // these keyword indexes, so their filters must not request a scoring
context either.
- return inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)
&&
- (get_parser_phrase_support_string_from_properties(properties) ==
- INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
- IndexReaderHelper::is_normalizer_only(properties));
+ // A keyword index keeps each value as one term, without positions or
norms, and its postings
+ // score with unit frequency and length whatever support_phrase says, as
an Elasticsearch
+ // keyword field does: over tags (x, x, y, z), tag MATCH 'x' scores idf(x)
= ln(2) for both
+ // matching rows, the same as SEARCH('tag:x'). MATCH statistics collect
these terms too.
+ if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)) {
+ return true;
+ }
Review Comment:
[P1] Exclude numeric indexes from keyword MATCH statistics
A non-analyzed index can also be numeric BKD. On a V2/V3 duplicate-key table
with a positional English index on `body` and an inverted index on `n INT`,
enable `enable_match_without_inverted_index=true` and
`enable_common_expr_pushdown=true`, then run `SELECT id, score() FROM t WHERE
body MATCH 'apple' AND CAST(n AS STRING) MATCH '1' ORDER BY score() DESC LIMIT
10`. Execution intentionally leaves the numeric-to-string MATCH as a scalar
filter, but collection unwraps the cast to `n` and this new branch admits its
BKD index. `process_segment()` consequently opens the BKD-only directory with
CLucene `IndexReader::open()` and aborts with `INVERTED_INDEX_CLUCENE_ERROR`;
the previous guard skipped it. Require a compatible text reader before
collecting keyword terms and cover this mixed score/filter case.
##########
be/src/storage/index/inverted/reader/predicate_collector.cpp:
##########
@@ -365,15 +383,14 @@ Status MatchPredicateCollector::collect(RuntimeState*
state, const TabletSchemaS
const auto* index_meta = DORIS_TRY(
select_index_meta(candidates.index_metas, candidates.field_type,
query_type,
analyzer_ctx->analyzer_key,
analyzer_ctx->legacy_analyzer_key));
- if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta->properties())
||
- !IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
+ if (!IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
Review Comment:
[P1] Keep statistics for different indexes on one column separate
With both a keyword index and a positional English index on `name`, `SELECT
id, score() FROM t WHERE name MATCH 'apple' USING ANALYZER english AND name
MATCH 'apple' USING ANALYZER none ORDER BY score() DESC LIMIT 10` now fails
during statistics collection. Both leaves use the same UID-based
`collect_infos` key, so the second reaches `validate_same_physical_index()` and
returns "Scoring predicates for one field select different inverted indexes".
The previous guard skipped the keyword leaf, so this query worked. Carry
physical-index identity through collection and scorer lookup before admitting
both leaves, and add a same-column mixed-analyzer score regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
