airborne12 commented on code in PR #67538:
URL: https://github.com/apache/doris/pull/67538#discussion_r3988885118
##########
be/src/storage/index/snii/reader/logical_index_reader.h:
##########
@@ -176,11 +176,23 @@ class LogicalIndexReader {
const format::SectionRefs& section_refs() const { return
core_.section_refs; }
const format::StatsBlock& stats() const { return core_.stats; }
+ // Bounds on the df of this index's most common terms, resident since the
segment was
+ // opened. Lets a caller decide a term is too common to be worth reading
without issuing
+ // the dictionary read that would tell it exactly how common. Empty on
indexes written
+ // before the digest existed, in which case there is no bound and df must
be read.
+ const format::HighDfTerms& high_df_terms() const { return
core_.high_df_terms; }
Review Comment:
Confirmed and fixed in fd3990cf5cfeec2e972f886790005363947f14ce.
`memory_usage()` now adds both digest vectors' capacities using the same
saturating arithmetic as the norms charge.
Scale of the under-count, for the record: the digest is capped at `min(4096,
max(64, term_count / 64))` entries of 12 bytes, so it is about 48 KiB per
cached segment index at the writer's cap and proportionally less on a small
vocabulary -- real, but bounded, and it only ever made the cache hold more than
it thought, never less.
Test: `SniiStopGram.TheSearcherCacheChargeIncludesTheResidentDigest` builds
two readers over one corpus, one with a gram scheme and one without, and
requires the difference in charge to cover the digest the first one carries.
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -651,6 +737,12 @@ Status SniiIndexReader::_query(const IndexQueryContextPtr&
context, const std::s
const ::doris::snii::reader::LogicalIndexReader* logical_reader = nullptr;
RETURN_IF_ERROR(_get_logical_reader(context, &searcher_cache_handle,
&uncached_reader,
&logical_reader));
+ if (analyzed_query && logical_reader->gram_scheme().has_value() &&
Review Comment:
Confirmed and fixed in 23b73103e5c04d36a88a29c6e9ae4358c0e26288.
Reproduced on a live cluster: a table indexed by a legacy ngram tokenizer
(`min_gram=2, max_gram=2`, no `mode`), dropped without FORCE, its policy
recreated in gram mode, then `RECOVER TABLE`. `MATCH_ANY 'abcdef'` then
returned `[]` with the index on and `[1 2]` with it off. (First attempt at this
repro passed spuriously because the script only waited for `TOKENIZE` to
succeed; the replacement policy has the same name, so the old analyzer answers
until the next heartbeat. Polling `JSON_LENGTH(TOKENIZE(...))` for the expected
token count made it deterministic.)
The guard now compares the two optionals instead of two schemes, so an
absent persisted scheme is a mismatch like any other whenever the current
analyzer has one. Test: `test_gram_policy_recovery` gained a
legacy-ngram-to-gram recovery phase running MATCH_ANY / MATCH_ALL / LIKE with
the index on and off.
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -156,6 +164,23 @@ bool uses_phrase_frequency_scoring(InvertedIndexQueryType
query_type,
query_type == InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY);
}
+// Query types whose terms come out of the current analyzer. On a gram-family
index those
+// terms are grams, and they only mean what the segment's own grams mean when
both were cut
+// by the same scheme; a gram query compiles against the segment's scheme
itself and a raw
+// pattern query never analyzes, so neither is affected.
Review Comment:
Confirmed and fixed in 23b73103e5c04d36a88a29c6e9ae4358c0e26288.
Reproduced on a live cluster on the dense-3 segment recovered under a
dense-4 analyzer: `MATCH_REGEXP '^abcd$'` returned `[]` with the index on and
`[1 2]` with it off, while `MATCH_ANY 'abcd'` agreed on both arms -- which
isolates the gap to this subtype rather than to the fence in general.
`MATCH_REGEXP_QUERY` now joins the analyzed subtypes in
`analyzes_query_terms`, which is both the scheme fence and the predicate that
keeps such a query out of the scheme-blind result cache, so the cache half of
the report is covered by the same change. The comment there now says why a raw
pattern still counts as analyzer-dependent. Test: `test_gram_policy_recovery`
runs `MATCH_REGEXP '^abcd$'` index-on and index-off on the recovered table and
on the mixed dense-3 / dense-4 table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
