github-actions[bot] commented on code in PR #67219:
URL: https://github.com/apache/doris/pull/67219#discussion_r3889591680
##########
fe/fe-catalog/src/main/java/org/apache/doris/catalog/Index.java:
##########
@@ -75,15 +75,21 @@ public Index(long indexId, String indexName, List<String>
columns,
this.comment = comment;
if (indexType == IndexType.INVERTED) {
if (this.properties != null && !this.properties.isEmpty()) {
- if
(this.properties.containsKey(InvertedIndexProperties.INVERTED_INDEX_PARSER_KEY)
- ||
this.properties.containsKey(InvertedIndexProperties.INVERTED_INDEX_PARSER_KEY_ALIAS)
- ||
this.properties.containsKey(InvertedIndexProperties.INVERTED_INDEX_ANALYZER_NAME_KEY)
- ||
this.properties.containsKey(InvertedIndexProperties.INVERTED_INDEX_NORMALIZER_NAME_KEY))
{
- String supportPhraseKey = InvertedIndexProperties
- .INVERTED_INDEX_SUPPORT_PHRASE_KEY;
+ String supportPhraseKey =
InvertedIndexProperties.INVERTED_INDEX_SUPPORT_PHRASE_KEY;
+ if (isTokenizedInvertedIndex(this.properties)) {
if (!this.properties.containsKey(supportPhraseKey)) {
this.properties.put(supportPhraseKey, "true");
}
+ } else {
+ // No analyzer on either side, so the query string is
never split
+ // either: InvertedIndexAnalyzer::get_analyse_result
returns the
+ // whole search string as ONE term, and every phrase
variant takes
+ // its terms from there. A single-term phrase is a term
query, so
+ // no query against this index can observe a position.
Drop the
+ // option instead of carrying it down to the BE, where it
would ask
+ // for position data nobody can read and make the index
look
+ // scoreable to
IndexReaderHelper::is_need_similarity_score.
+ this.properties.remove(supportPhraseKey);
Review Comment:
`support_phrase` controls term frequencies as well as positions, so
`should_analyzer()` is too broad a condition for removing it. For a reachable
`ARRAY<STRING>` index with `parser=none`, the legacy writer adds every element
as another field in the same document; `["x", "x"]` therefore has tf=2 while
`["x"]` has tf=1. SEARCH's scoring collector admits this keyword index, and its
query-v2 `TermQuery` reads that frequency for BM25. Removing the property here
makes new V1/V2/V3 postings omit frequencies, turning those rows into a score
tie and potentially changing `ORDER BY score() ... LIMIT` results. Please
preserve the frequency-bearing representation for this case (or separate
frequency retention from phrase positions) and add a physical repeated-ARRAY
score regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
