airborne12 commented on code in PR #67917:
URL: https://github.com/apache/doris/pull/67917#discussion_r4000964962
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java:
##########
@@ -129,8 +129,13 @@ public List<String> getShowInfo() {
ImmutableSet.of("common_grams");
public boolean isInvalid() {
- return type == IndexPolicyTypeEnum.TOKEN_FILTER
+ boolean hasUnsupportedTokenFilter = type ==
IndexPolicyTypeEnum.TOKEN_FILTER
&& properties != null
&&
LEGACY_UNSUPPORTED_TOKEN_FILTER_TYPES.contains(properties.get(PROP_TYPE));
+ boolean hasInvalidNgramTokenizer = type ==
IndexPolicyTypeEnum.TOKENIZER
+ && properties != null
+ && "ngram".equals(properties.get(PROP_TYPE))
+ && !NGramTokenizerValidator.isValidPolicy(properties);
Review Comment:
Fixed in 035a04947b3. Newly created ngram policies now persist
max_ngram_diff=1 when omitted, which is an explicit compatibility marker that
old metadata cannot contain. FE replay validation and BE factory reconstruction
enforce the 1024 cap only when this marker is present, so marker-less policies
accepted by the previous release retain their former
positive/order/default-difference behavior during BE-first upgrades. Coverage
includes an IndexPolicy serialization round trip at min_gram=max_gram=2048,
replay of its dependent analyzer through validateAnalyzerExists, persistence of
the marker on new creation, rejection of marked oversized metadata, and
successful BE construction of the legacy 2048 tokenizer. Final validation:
focused BE compatibility cases 3/3, AnalyzerIdentityBuilderTest plus
PolicyValidatorTests 35/35, full ASAN BE and FE builds, clang-format 16,
changed-line clang-tidy, full English gate, and merge-tree Checkstyle/format
preflight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
