airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4056034188
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +310,81 @@ private static String resolveTokenFilterIdentity(String
filterList) {
* IMPORTANT: Order is preserved because filter order is semantically
significant.
*/
private static String resolveCharFilterIdentity(String filterList) {
+ return resolveCharFilterIdentity(filterList, false);
+ }
+
+ private static String resolveCharFilterIdentity(String filterList, boolean
lowercaseIk) {
if (Strings.isNullOrEmpty(filterList)) {
return "";
}
- StringBuilder sb = new StringBuilder();
+ ArrayDeque<String> identities = new ArrayDeque<>();
String[] filters = filterList.split(",\\s*");
// DO NOT sort - filter order is semantically significant
- for (int i = 0; i < filters.length; i++) {
- String filter = filters[i].trim();
- if (i > 0) {
- sb.append(",");
+ for (int i = filters.length - 1; i >= 0; --i) {
+ String filter = resolveComponentIdentity(filters[i].trim(),
IndexPolicyTypeEnum.CHAR_FILTER, lowercaseIk);
+ if (Strings.isNullOrEmpty(filter)) {
+ continue;
}
+ identities.addFirst(filter);
+ // An earlier replacement can change the input of a later filter.
+ lowercaseIk = false;
Review Comment:
Fixed in `51d2e076add`.
The reverse character-filter walk now carries a case-folding fact through an
unrestricted default `icu_normalizer` (`nfkc_cf`). It does not propagate that
fact through composition-only normalization or a normalizer restricted by
`unicode_set_filter`, so only the proven redundant replacement is absorbed.
Before the fix, the new focused FE test showed different identities for
fold-only and lower-then-fold pipelines. After the fix, that method passed and
the complete `AnalyzerIdentityBuilderTest` class passed 22/22. The regression
suite also covers both CREATE and ALTER rejection of the equivalent second
index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
