airborne12 opened a new pull request, #68774:
URL: https://github.com/apache/doris/pull/68774
### What problem does this PR solve?
Issue Number: close #67916
Related PR: #67917
Problem Summary:
Cherry-pick of #67917 (commit 6a79f954e26, a squash merge) to branch-4.2.
Custom ngram tokenizers can now set `max_ngram_diff` (0 to 255, default 1)
instead of the hard-coded difference of 1. FE and BE validate the same range,
and newly created policies cap absolute gram sizes at 1024. A persisted
`max_ngram_diff` marker keeps marker-less policies from before the change
valid. `max_ngram_diff` does not change the analyzer identity.
branch-4.2 also has the gram `mode` tokenizer, which the source change does
not know about. Adaptations:
- `max_ngram_diff` and the size limits apply to the legacy (no `mode`) path
only.
- `max_ngram_diff` together with `mode` is rejected.
- The compatibility marker is added only to new legacy ngram policies.
### Release note
Allow custom ngram tokenizers to configure the maximum difference between
max_gram and min_gram with max_ngram_diff values from 0 through 255.
### Check List (For Author)
- Test
- [ ] Regression test (the case is included; it was not run locally and
is left to CI)
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. New custom ngram tokenizers can opt into a wider gram-size
range (difference up to 255, absolute size up to 1024).
- Does this need documentation?
- [ ] No.
- [x] Yes. Same documentation as the source PR.
### Validation
- FE: `run-fe-ut.sh --run PolicyValidatorTests,AnalyzerIdentityBuilderTest`:
117 tests, 0 failures; Checkstyle passed.
- BE: `NGramTokenizerTest.*`: 21/21 passed (includes the 7 new tests), ASAN.
- clang-format 16 and Checkstyle merge-tree preflight: PASS.
- Local-environment notes (not part of this PR): `origin/branch-4.2`
currently declares thrift field id 229 twice in `PaloInternalService.thrift`
(`enable_paimon_rust_reader`, `enable_prune_nested_column`), so a clean
branch-4.2 build fails; I bumped one id locally only to run the tests. For the
BE UT I also stubbed the unrelated paimon_rust sources locally and linked with
`--unresolved-symbols=ignore-all`. None of this is in the commit.
### Hunk audit (against #67917)
| File | Hunk | Result |
|---|---|---|
| ngram_tokenizer.cpp | `@@ NGramTokenizer::init` buffer size |
Ported(verbatim) |
| ngram_tokenizer.h | trailing newline | Ported (4.2 keeps
`compile_check_end.h`) |
| ngram_tokenizer_factory.cpp | `initialize` limits and diff check, newline
| Ported(verbatim); the 4.2 gram branch returns earlier, so these apply to the
legacy path only |
| ngram_tokenizer_factory.h | `MAX_NGRAM_DIFF`/`MAX_NGRAM_SIZE` constants |
Ported(verbatim) |
| ngram_tokenizer_test.cpp | 7 new tests | Ported; 4.2's
`RejectsMalformedUtf8AndCanBeReset` kept |
| NGramTokenizerValidator.java | `ALLOWED_PROPS`, constants,
`isValidPolicy`, size/diff checks | Adapted: 4.2 `mode` props kept,
`max_ngram_diff` rejected with `mode` |
| IndexPolicy.java | `@@ isInvalid` | Ported (uses `isValidPolicy`; same
behavior as 4.2's try/catch) |
| IndexPolicyMgr.java | `@@ validateReferencedComponentsUsableLocked` | N-A:
4.2 already has it (tokenizer `isInvalid` check present) |
| IndexPolicyMgr.java | `@@ createIndexPolicy` marker | Adapted: only for
non-`mode` ngram |
| IndexPolicyMgr.java | `@@ validateReferencedPolicy` isInvalid check | N-A:
already present in 4.2 |
| IndexPolicyMgr.java | `@@ warnIfUnsupported` message | Ported |
| AnalyzerIdentityBuilder.java | `PROP_MAX_NGRAM_DIFF`, `isInvalid`
identity, `remove(max_ngram_diff)` | N-A: all three already present in 4.2
(verified with `git grep` on `origin/branch-4.2`) |
| AnalyzerIdentityBuilderTest.java | 3 tests | Ported; 4.2's
`testNgramDefaultBoundsDoNotChangeAnalyzerIdentity` kept |
| PolicyValidatorTests.java | difference/size/marker tests | Ported; 4.2
gram-mode tests kept; 1 new test for `max_ngram_diff` + `mode` |
| regression-test .groovy/.out | new files | Ported(verbatim) |
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
