airborne12 opened a new pull request, #68561: URL: https://github.com/apache/doris/pull/68561
### What problem does this PR solve? Related PR: #67917 (https://github.com/apache/doris/pull/67917) Target: `branch-4.1` Source merged commit: `6a79f954e2617c807173aad9796cfd6138c14055` Backport commit: `b668aca415456108f21c631baf46b4e20b2665bf` Problem Summary: Backport configurable `max_ngram_diff` for custom ngram tokenizers. The default stays at 1, explicit values must be ASCII integers from 0 through 255, and new policies limit `min_gram` and `max_gram` to 1024. Marker-less persisted policies retain the absolute sizes accepted before this change. Invalid replayed policies cannot block valid replacements. `max_ngram_diff` is excluded from analyzer identity because it controls admission, not emitted tokens. The 4.1 adaptation retains its existing policy ID allocation and omits the source branch's legacy `common_grams` handling, which is absent in 4.1. A serialization helper was added to the target's FE tests because the source branch already had it. The source-to-backport range-diff maps one squash commit to one backport commit; the hunk audit below records each source change. This PR overlaps with #68550 in eight files. A local merge-tree simulation predicts six content conflicts if both independent backports are merged without updating the later branch. Merge #67917's backport before #67918's backport and resolve the latter against the updated 4.1 branch. ### Release note Allow custom ngram tokenizers to configure the maximum difference between `max_gram` and `min_gram` with `max_ngram_diff` values from 0 through 255. ### Check List (For Author) - Test - [x] FE build and Checkstyle passed with zero violations. - [x] FE `PolicyValidatorTests` 25/25 and `AnalyzerIdentityBuilderTest` 8/8 passed. - [x] ASAN BE build passed; clang-format 16 checked all five changed C++ files, and changed-line clang-tidy passed. - [x] ASAN BE `NGramTokenizerTest` passed 19/19. - [x] Regression `test_ngram_max_diff_custom_analyzer` passed 1/1 after runner-generated output and a normal comparison run. - Behavior changed: - [x] Yes. New custom ngram policies can opt into a wider gram-size difference; old persisted policies retain their prior absolute-size behavior. - Does this need documentation? - [x] Yes. The custom analyzer documentation should describe `max_ngram_diff`, its 0 through 255 range, and the 1024 absolute-size limit for newly created policies. The source PR did not include a documentation change. The local BE link used a task-owned compatible Lance dependency copy with an uncommitted shim for missing Lance symbols; this work did not validate Lance behavior. The dependency copy and shim are outside the PR diff. The isolated BE regression instance disabled Arrow Flight SQL after its local flight port bind failed; the tested NGram path does not use Arrow Flight SQL. The final range-diff reports `1: 6a79f954e26 ! 1: b668aca4154`: one source squash commit maps to one backport commit. The `!` records branch-4.1 adaptations in the ngram header, policy replay validation, policy ID allocation, and the FE test helper; the table below explains each changed source hunk. ### Source hunk audit # PR #67917 to branch-4.1 source hunk audit Source squash commit: `6a79f954e2617c807173aad9796cfd6138c14055`. Every row below refers to one source `@@` hunk unless the final file bytes match exactly. ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | `@@ -106 +106,2 @@ void NGramTokenizer::init(int32_t min_gram, int32_t max_gram, bool edges_only) {` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | `@@ -78 +78 @@ private:` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer_factory.h` | Source hunk | Status | Evidence | |---|---|---| | `@@ -28,0 +29,5 @@ public:` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | | `@@ -68 +73 @@ private:` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | ### `be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java` | Source hunk | Status | Evidence | |---|---|---| | `@@ -132 +132 @@ public class IndexPolicy implements Writable, GsonPostProcessable {` | Adapted | 4.1 has no legacy common_grams invalidation; retain its existing token-filter behavior and add ngram replay validation. | | `@@ -134,0 +135,5 @@ public class IndexPolicy implements Writable, GsonPostProcessable {` | Adapted | Invalid replayed ngram policies use NGramTokenizerValidator.isValidPolicy; no legacy filter status is combined. | ### `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java` | Source hunk | Status | Evidence | |---|---|---| | `@@ -113 +113 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Adapted | Call target-specific invalid-tokenizer check; 4.1 has no legacy common_grams validation. | | `@@ -120,4 +120,3 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | N-A | The 4.1 target has no common_grams filter handling or corresponding helper comment. | | `@@ -125 +124 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Adapted | Rename the new helper to reflect its tokenizer-only behavior in 4.1. | | `@@ -127,2 +126,13 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Ported | Reject analyzers that reference an invalid replayed ngram tokenizer. | | `@@ -189,2 +199,10 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Adapted | Copy properties and persist the compatibility marker while preserving 4.1 ID allocation before the manager lock. | | `@@ -339,0 +358,3 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Ported | Reject references to invalid replayed ngram policies. | | `@@ -670,4 +691,4 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable {` | Adapted | Log invalid replayed ngram policies in both image and edit-log replay; 4.1 lacked the prior common_grams warning helper. | ### `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/NGramTokenizerValidator.java` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java` | Source hunk | Status | Evidence | |---|---|---| | `@@ -19,0 +20 @@ package org.apache.doris.indexpolicy;` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | | `@@ -20,0 +22 @@ import org.apache.doris.common.DdlException;` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | | `@@ -23,0 +26,2 @@ import org.junit.jupiter.api.Test;` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | | `@@ -132,0 +137 @@ public class PolicyValidatorTests {` | Ported | Source hunk is present; other file differences come from the 4.1 baseline. | | `@@ -135,0 +141,124 @@ public class PolicyValidatorTests {` | Adapted | The 4.1 test file lacked the roundTrip helper inherited on master; it was added to run the replay compatibility tests. | ### `regression-test/data/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.out` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | ### `regression-test/suites/inverted_index_p0/analyzer/test_ngram_max_diff_custom_analyzer.groovy` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | Source and backport file bytes match. | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
