github-actions[bot] commented on code in PR #67917:
URL: https://github.com/apache/doris/pull/67917#discussion_r4000900770
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java:
##########
@@ -129,8 +129,13 @@ public List<String> getShowInfo() {
ImmutableSet.of("common_grams");
public boolean isInvalid() {
- return type == IndexPolicyTypeEnum.TOKEN_FILTER
+ boolean hasUnsupportedTokenFilter = type ==
IndexPolicyTypeEnum.TOKEN_FILTER
&& properties != null
&&
LEGACY_UNSUPPORTED_TOKEN_FILTER_TYPES.contains(properties.get(PROP_TYPE));
+ boolean hasInvalidNgramTokenizer = type ==
IndexPolicyTypeEnum.TOKENIZER
+ && properties != null
+ && "ngram".equals(properties.get(PROP_TYPE))
+ && !NGramTokenizerValidator.isValidPolicy(properties);
Review Comment:
[P1] Preserve n-gram policies that were valid before upgrade
This applies the new 1024 cap retroactively to replayed policies. In the
base code, a persisted tokenizer such as `min_gram=2048,max_gram=2048` passes
both FE and BE (the difference is 0), and its tokenizer buffer is only `4 *
2048 + 1024` `UChar32` slots, about 36 KiB. During the supported BE-first
rolling upgrade, new BEs start rejecting analyzer reconstruction for that
existing policy; after FE upgrade, this line also marks it invalid and
`validateAnalyzerExists` rejects every dependent analyzer, so existing indexes
become unusable solely because of the upgrade. This is distinct from the
existing absolute-allocation thread, which concerns bounding newly admitted
pathological values. Please grandfather or migrate policies that were valid
under the previous rules (with a matching BE reconstruction path), and add an
image/edit-log upgrade test for an existing analyzer/index above 1024.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
