github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4089823483
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -55,13 +56,65 @@ public class IndexPolicyMgr implements Writable,
GsonPostProcessable {
private final Map<Long, IndexPolicy> idToIndexPolicy = Maps.newHashMap();
// Keys are normalized to lowercase for case-insensitive lookup
private final Map<String, IndexPolicy> nameToIndexPolicy =
Maps.newHashMap();
+ // Legacy metadata can contain case-distinct names that share a normalized
key. Keep exact
+ // bindings separately so a saved analyzer continues to resolve its
original component.
+ private final transient Map<String, IndexPolicy> exactNameToIndexPolicy =
Maps.newHashMap();
/**
* Normalize policy name to lowercase for case-insensitive lookup.
* Policy names are case-insensitive in Doris.
*/
private static String normalizeKey(String name) {
- return name == null ? null : name.trim().toLowerCase();
+ return name == null ? null : name.trim().toLowerCase(Locale.ROOT);
+ }
+
+ private static String exactKey(String name) {
+ return name == null ? null : name.trim();
+ }
+
+ // Callers hold either the read or write lock. Prefer an exact legacy name
binding and
+ // retain normalized lookup only for interactive case-insensitive fallback.
+ private IndexPolicy getPolicyByNameLocked(String name) {
+ IndexPolicy exactPolicy = exactNameToIndexPolicy.get(exactKey(name));
+ return exactPolicy != null ? exactPolicy :
nameToIndexPolicy.get(normalizeKey(name));
+ }
+
+ // Callers hold either the read or write lock. BE dispatches a canonical
built-in analyzer, then an
+ // exact policy, then a built-in by normalized name; return a spelling
that reaches that built-in,
+ // or null for a policy.
+ private String resolveTopLevelBuiltinLocked(String name, Set<String>
builtins) {
+ String exactName = exactKey(name);
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(exactName) &&
builtins.contains(exactName)) {
Review Comment:
[P1] Reserve built-in analyzer names when validating normalizers. An exact
replayed `NORMALIZER` named `ik` passes this method because `ik` is not in
`BUILTIN_NORMALIZERS`, and `resolvePolicyNames` preserves that exact spelling.
BE then merges the `normalizer` property into `analyzer_name` and both analyzer
factories intercept canonical `ik` as the built-in before policy lookup, so the
accepted normalizer policy is never executed (the same applies to `standard`,
`english`, `none`, etc.). Reject these unreachable bindings or make FE model
BE's global built-in-analyzer precedence, and cover replayed collisions in
CREATE/ALTER.
##########
be/src/storage/index/inverted/inverted_index_iterator.cpp:
##########
@@ -164,8 +166,9 @@ Result<InvertedIndexReaderPtr>
InvertedIndexIterator::select_best_reader(
}
field_type = get_inverted_index_leaf_field_type(column_type);
}
- auto selection =
select_best_inverted_index_candidate(_selection_candidates, _key_to_entries,
- field_type,
query_type, normalized_key);
+ auto selection =
+ select_best_inverted_index_candidate(_selection_candidates,
_key_to_entries, field_type,
Review Comment:
[P1] Validate phrase support on the reader selected here.
`FunctionMatchBase::evaluate_inverted_index()` currently checks
`iter->get_reader(FULLTEXT)` first, and that helper returns the first full-text
candidate without considering `analyzer_key`; only afterward does this path
select the physical reader that executes the query. With a first `ik_smart`
index and a separately selectable default `ik` index, opposite `support_phrase`
settings either reject a valid selected index or let a phrase query run on an
index that stored no positions, solely by index order. Move the capability
check after analyzer-aware selection (or return the selected reader for the
preflight), and cover both orderings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
