airborne12 commented on code in PR #67917:
URL: https://github.com/apache/doris/pull/67917#discussion_r4000824240
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -177,6 +179,11 @@ private static String resolveComponentIdentity(String
name, IndexPolicyTypeEnum
// Build identity from sorted properties
TreeMap<String, String> sortedProps = new TreeMap<>(props);
+ if (expectedType == IndexPolicyTypeEnum.TOKENIZER
+ && "ngram".equals(sortedProps.get(IndexPolicy.PROP_TYPE)))
{
+ // This setting only limits policy creation; it does not
change emitted tokens.
+ sortedProps.remove(PROP_MAX_NGRAM_DIFF);
Review Comment:
Fixed in 8de9abb95d0. Replayed ngram tokenizer policies that fail current
validation are now marked unusable: analyzer validation rejects references to
them, and AnalyzerIdentityBuilder assigns a policy-id-specific invalid fallback
before runtime-property canonicalization. A replay test covers the legacy
invalid 1..8 policy, proves it cannot be used, and verifies that a valid
max_ngram_diff=7 replacement has a distinct identity. Valid policies still
retain ceiling-insensitive identity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
