airborne12 opened a new pull request, #68550: URL: https://github.com/apache/doris/pull/68550
### What problem does this PR solve? Related PR: #67918 (https://github.com/apache/doris/pull/67918) Source merged commit: `ef72b93c673b0f99827684bfe97657b39db01989` Backport commit: `b24f0ea30f029aeade5dcad75ed7d09594cd57ee` Target: `branch-4.1` Problem Summary: Backport IK smart and max-word tokenizer support for custom inverted-index analyzers. The patch also carries source-byte offset provenance through tokenizer and filter composition, exact saved-policy identity, physical reader selection, and Status conversion at analyzer boundaries. The 4.1 FE code lives in `fe-core`, while several BE test and reader interfaces differ from master. The backport maps those interfaces, keeps 4.1's existing NGram bounds because `max_ngram_diff` is absent, and updates old tests for the merged analyzer identity rules. The source-to-backport `git range-diff` maps one squash commit to one backport commit (`ef72b93c673 ! b24f0ea30f0`); the table below records every source hunk and its 4.1 treatment. ### Release note Custom inverted-index analyzers support `ik_smart` and `ik_max_word` tokenizers. ### Check List (For Author) - Test - [x] Regression test: `test_analyzer_identity_semantics`, `test_ik_custom_analyzer`, and `test_analyzer_malformed_utf8_write` passed on an isolated ASAN FE/BE cluster (3/3). - [x] Unit Test: BE ASAN, 437/437 tests in 16 affected suites; all 95 added BE tests were discovered and executed. FE `PolicyValidatorTests` 34/34 and `AnalyzerIdentityBuilderTest` 62/62 passed after the final FE change. - [x] Build: `./build.sh --be-java-extensions`, `./build.sh --be -j8`, and `./build.sh --fe` passed. The FE build reported zero Checkstyle violations. - [x] Style: clang-format 16 checked all 79 changed C++ files; clang-tidy reported no changed-line warnings after the 4.1 test adaptations. - [x] English audit: 100 source files passed; 45 exact exceptions are Unicode input or expected-token literals in tests. No descriptive comments, diagnostics, or messages were exempted. - Behavior changed: - [x] Yes. Custom IK tokenization, analyzer identity resolution, and source offsets now work across DDL, indexing, and MATCH queries. - Does this need documentation? - [x] No. Existing analyzer configuration syntax is reused. Local BE linking needed an ignored compatibility shim because the available pinned Lance archive lacks newer symbols and host Rust subprocess output capture prevents rebuilding it. The shim is outside this commit; no Lance behavior was tested. A mixed-version upgrade cluster was not run. The regression runner generates a required blank separator at EOF in `test_analyzer_identity_semantics.out`, which standard `git diff --check` reports as a blank-line warning; the generated file was left unchanged and all other whitespace checks pass. ### Source hunk audit # PR 67918 to branch-4.1 hunk audit Source: `ef72b93c673b0f99827684bfe97657b39db01989`. Each `@@` header is from the source diff. `Ported (verbatim)` summarizes a byte-identical source file. ### `be/src/exprs/function/function_search.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -304,0 +305,23 @@ Status FunctionSearch::evaluate_inverted_index_with_search_param( | Ported | | ### `be/src/exprs/function/function_search.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/exprs/function/match.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -65,8 +65 @@ Status FunctionMatchBase::evaluate_inverted_index( | Ported | | ### `be/src/exprs/function/variant_inverted_index_search.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -68 +68 @@ void add_search_binding_diagnostic(const std::shared_ptr<IndexQueryContext>& con | Ported | | | @@ -92,0 +93,14 @@ InvertedIndexAnalyzerCtxSPtr build_analyzer_context( | Ported | | | @@ -418,2 +432,6 @@ Status FieldReaderResolver::resolve_with_analyzer_context(const std::string& fie | N-A | The 4.1 FieldReaderResolver has no resolve_with_analyzer_context or SNII reader; the Status-converting helper remains in build_search_analyzer_context and its test. | ### `be/src/exprs/function/variant_inverted_index_search.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -104,0 +105,5 @@ struct FieldReaderBinding { | Ported | | ### `be/src/exprs/vmatch_predicate.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -61,2 +61,4 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) { | Ported | | | @@ -84,2 +86,4 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) { | Ported | | | @@ -86,0 +91,18 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) { | Ported | | ### `be/src/runtime/index_policy/index_policy_mgr.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/runtime/index_policy/index_policy_mgr.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/analysis_factory_mgr.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/analyzer/analyzer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -188 +188,2 @@ AnalyzerProviderPtr InvertedIndexAnalyzer::create_analyzer_provider( | Ported | | | @@ -189,0 +191,3 @@ AnalyzerProviderPtr InvertedIndexAnalyzer::create_analyzer_provider( | Ported | | | @@ -190,0 +195,3 @@ AnalyzerProviderPtr InvertedIndexAnalyzer::create_analyzer_provider( | Ported | | | @@ -199,2 +206,2 @@ AnalyzerProviderPtr InvertedIndexAnalyzer::create_analyzer_provider( | Ported | | ### `be/src/storage/index/inverted/analyzer/analyzer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/analyzer/custom_analyzer.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/analyzer/ik/IKTokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20,7 @@ | Ported | | | @@ -21,0 +29,37 @@ namespace doris::segment_v2 { | Ported | | | @@ -34 +78 @@ Token* IKTokenizer::next(Token* token) { | Ported | | | @@ -37,3 +81,45 @@ Token* IKTokenizer::next(Token* token) { | Adapted | The UTF-8 prefix call is present at IKTokenizer.cpp:105; clang-format wraps the assignment differently. | | @@ -42,0 +129,9 @@ Token* IKTokenizer::next(Token* token) { | Ported | | | @@ -43,0 +139 @@ void IKTokenizer::reset(lucene::util::Reader* reader) { | Ported | | | @@ -44,0 +141 @@ void IKTokenizer::reset(lucene::util::Reader* reader) { | Ported | | | @@ -47 +144,6 @@ void IKTokenizer::reset(lucene::util::Reader* reader) { | Ported | | | @@ -54 +156,5 @@ void IKTokenizer::reset(lucene::util::Reader* reader) { | Ported | | | @@ -63 +169 @@ void IKTokenizer::reset(lucene::util::Reader* reader) { | Ported | | ### `be/src/storage/index/inverted/analyzer/ik/IKTokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/analyzer/ik/core/AnalyzeContext.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -187 +187 @@ void AnalyzeContext::markBufferOffset() { | Ported | | | @@ -297 +297 @@ void AnalyzeContext::outputSingleCJK(size_t index) { | Ported | | ### `be/src/storage/index/inverted/char_filter/char_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/char_filter/icu_normalizer_char_filter.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/char_filter/icu_normalizer_char_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/inverted_index_iterator.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -23,0 +24 @@ | Ported | | | @@ -72,2 +73,4 @@ Status InvertedIndexIterator::read_from_index(const IndexParam& param) { | Ported | | | @@ -77,0 +81,9 @@ Status InvertedIndexIterator::read_from_index(const IndexParam& param) { | Ported | | | @@ -153 +165 @@ Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_best_reader( | Ported | | | @@ -167,2 +179,3 @@ Result<InvertedIndexReaderPtr> InvertedIndexIterator::select_best_reader( | Ported | | ### `be/src/storage/index/inverted/inverted_index_iterator.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -63 +63 @@ public: | Ported | | | @@ -94 +94 @@ private: | Ported | | ### `be/src/storage/index/inverted/inverted_index_parser.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20,4 @@ | Ported | | | @@ -20,0 +25 @@ | Ported | | | @@ -214,4 +219 @@ std::string normalize_analyzer_key(std::string_view analyzer) { | Ported | | | @@ -220,5 +222,24 @@ std::string normalize_analyzer_key(std::string_view analyzer) { | Ported | | | @@ -225,0 +247,3 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -227,9 +251,12 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -236,0 +264,2 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -238,2 +267,10 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -241 +278,4 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -248,10 +288,2 @@ std::string build_analyzer_key_from_properties( | Ported | | | @@ -261 +293,3 @@ AnalyzerConfig AnalyzerConfigParser::parse(const std::string& analyzer_name, | Ported | | | @@ -263,2 +296,0 @@ AnalyzerConfig AnalyzerConfigParser::parse(const std::string& analyzer_name, | Ported | | | @@ -266,4 +298,5 @@ AnalyzerConfig AnalyzerConfigParser::parse(const std::string& analyzer_name, | Ported | | ### `be/src/storage/index/inverted/inverted_index_parser.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -115 +115 @@ const std::string INVERTED_INDEX_PARSER_FIELD_PATTERN_KEY = "field_pattern"; | Ported | | | @@ -121,2 +121 @@ struct InvertedIndexAnalyzerCtx { | Ported | | | @@ -124,0 +124,3 @@ struct InvertedIndexAnalyzerCtx { | Ported | | | @@ -198,2 +200,2 @@ std::string get_analyzer_name_from_properties(const std::map<std::string, std::s | Ported | | | @@ -207 +209 @@ struct AnalyzerConfig { | Ported | | | @@ -215,3 +217 @@ struct AnalyzerConfig { | Ported | | | @@ -220 +220 @@ public: | Ported | | | @@ -224,4 +224,4 @@ public: | Ported | | | @@ -229,2 +229,2 @@ public: | Ported | | ### `be/src/storage/index/inverted/inverted_index_query_type.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/inverted_index_reader.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -486,0 +487,3 @@ Status FullTextIndexReader::query(const IndexQueryContextPtr& context, | Ported | | ### `be/src/storage/index/inverted/inverted_index_selector.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/inverted_index_selector.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/inverted_index_writer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -256,0 +257,4 @@ Status InvertedIndexColumnWriter<field_type>::add_document() { | Ported | | | @@ -331,0 +336,3 @@ Status InvertedIndexColumnWriter<field_type>::new_inverted_index_field(const cha | Ported | | | @@ -453,10 +460,23 @@ Status InvertedIndexColumnWriter<field_type>::add_array_values(size_t field_size | Ported | | ### `be/src/storage/index/inverted/inverted_index_writer.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -78,0 +79,9 @@ public: | Ported | | ### `be/src/storage/index/inverted/similarity/predicate_collector.cpp` → `be/src/storage/predicate_collector.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -44 +44 @@ namespace { | Ported | | | @@ -64,2 +64,20 @@ InvertedIndexAnalyzerCtx analyzer_context_from_properties( | Adapted | The collector lives at be/src/storage/predicate_collector.cpp in 4.1; analyzer_context_from_properties catches provider exceptions and analyze_plain_query catches token-stream exceptions. | | @@ -67,5 +85,13 @@ std::vector<TermInfo> analyze_plain_query(const std::string& value, | Ported | | | @@ -129 +155,2 @@ Result<const TabletIndex*> select_index_meta(const std::vector<const TabletIndex | Adapted | The 4.1 select_index_meta takes an explicit legacy key and passes both keys to select_best_inverted_index_candidate. | | @@ -147 +174,2 @@ Result<const TabletIndex*> select_index_meta(const std::vector<const TabletIndex | Ported | | | @@ -379,2 +407,3 @@ Status MatchPredicateCollector::collect(RuntimeState* state, const TabletSchemaS | Adapted | The 4.1 MATCH collector selects the runtime analyzer index, then analyzes with VMatchPredicate::query_analyzer_ctx and propagates Status. | | @@ -389 +418 @@ Status MatchPredicateCollector::collect(RuntimeState* state, const TabletSchemaS | Adapted | The 4.1 MATCH collector selects the runtime analyzer index, then analyzes with VMatchPredicate::query_analyzer_ctx and propagates Status. | | @@ -519 +548,5 @@ Status SearchPredicateCollector::collect_from_leaf(const TSearchClause& clause, | Adapted | The 4.1 SEARCH collector keeps its per-index loop; analyze_plain_query(value, properties) calls analyzer_context_from_properties and propagates errors as Result. | | @@ -526 +559 @@ Status SearchPredicateCollector::collect_from_leaf(const TSearchClause& clause, | Adapted | The 4.1 SEARCH collector keeps its per-index loop; analyze_plain_query(value, properties) calls analyzer_context_from_properties and propagates errors as Result. | | @@ -532 +565 @@ Status SearchPredicateCollector::collect_from_leaf(const TSearchClause& clause, | Adapted | The 4.1 SEARCH collector keeps its per-index loop; analyze_plain_query(value, properties) calls analyzer_context_from_properties and propagates errors as Result. | ### `be/src/storage/index/inverted/similarity/predicate_collector.h` → `be/src/storage/predicate_collector.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -20,0 +21 @@ | Ported | | | @@ -33,0 +35,7 @@ namespace doris { | Ported | | ### `be/src/storage/index/inverted/token_filter/ascii_folding_filter.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -24,0 +25,20 @@ namespace doris::segment_v2::inverted_index { | Ported | | | @@ -28,0 +49,2 @@ Token* ASCIIFoldingFilter::next(Token* t) { | Ported | | | @@ -45,0 +68,12 @@ Token* ASCIIFoldingFilter::next(Token* t) { | Ported | | | @@ -57,0 +92,29 @@ void ASCIIFoldingFilter::reset() { | Ported | | | @@ -2020 +2083 @@ int32_t ASCIIFoldingFilter::fold_to_ascii(const char* in, int32_t input_pos, cha | Ported | | ### `be/src/storage/index/inverted/token_filter/ascii_folding_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/icu_normalizer_filter.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/icu_normalizer_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/lower_case_filter.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -68,0 +69,37 @@ public: | Ported | | | @@ -80,0 +118,2 @@ public: | Ported | | | @@ -133,0 +173,7 @@ public: | Ported | | | @@ -138,2 +183,0 @@ public: | Ported | | | @@ -140,0 +185,16 @@ private: | Ported | | | @@ -142,0 +203,5 @@ private: | Ported | | | @@ -146 +211 @@ using LowerCaseFilterPtr = std::shared_ptr<LowerCaseFilter>; | Ported | | ### `be/src/storage/index/inverted/token_filter/pinyin_filter.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/pinyin_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/pinyin_filter_factory.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/token_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_filter/word_delimiter_filter.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20,3 @@ | Ported | | | @@ -35 +38 @@ WordDelimiterFilter::WordDelimiterFilter(const TokenStreamPtr& in, | Ported | | | @@ -42,0 +46 @@ WordDelimiterFilter::WordDelimiterFilter(const TokenStreamPtr& in, | Ported | | | @@ -52,0 +57,3 @@ Token* WordDelimiterFilter::next(Token* t) { | Ported | | | @@ -63,0 +71,3 @@ Token* WordDelimiterFilter::next(Token* t) { | Ported | | | @@ -83,0 +94,3 @@ Token* WordDelimiterFilter::next(Token* t) { | Ported | | | @@ -118,0 +132,6 @@ Token* WordDelimiterFilter::next(Token* t) { | Ported | | | @@ -135,0 +155,5 @@ Token* WordDelimiterFilter::next(Token* t) { | Ported | | | @@ -171,0 +196 @@ void WordDelimiterFilter::reset() { | Ported | | | @@ -177,0 +203,61 @@ void WordDelimiterFilter::reset() { | Ported | | | @@ -202,0 +289,4 @@ void WordDelimiterFilter::buffer() { | Ported | | | @@ -210,0 +301,79 @@ void WordDelimiterFilter::generate_part(bool is_single_word) { | Ported | | | @@ -249 +418 @@ bool WordDelimiterFilter::should_generate_parts(int32_t word_type) { | Ported | | ### `be/src/storage/index/inverted/token_filter/word_delimiter_filter.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/token_stream.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -20,0 +21 @@ | Ported | | | @@ -21,0 +23,2 @@ | Ported | | | @@ -22,0 +26 @@ | Ported | | | @@ -23,0 +28 @@ | Ported | | | @@ -38,0 +44,30 @@ using TokenStreamPtr = std::shared_ptr<TokenStream>; | Ported | | | @@ -61,0 +97,14 @@ public: | Ported | | ### `be/src/storage/index/inverted/tokenizer/basic/basic_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -53,0 +54,5 @@ Token* BasicTokenizer::next(Token* token) { | Ported | | | @@ -126 +131 @@ void BasicTokenizer::cut() { | Ported | | ### `be/src/storage/index/inverted/tokenizer/char/char_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -128,0 +129,3 @@ Token* CharTokenizer::next(Token* token) { | Ported | | | @@ -139 +142 @@ void CharTokenizer::reset() { | Ported | | ### `be/src/storage/index/inverted/tokenizer/empty/empty_tokenizer_factory.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -39,0 +40,3 @@ public: | Ported | | | @@ -76 +79 @@ public: | Ported | | ### `be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -20,0 +21 @@ | Ported | | | @@ -24,0 +26,3 @@ | Ported | | | @@ -56,0 +61,4 @@ Token* ICUTokenizer::next(Token* token) { | Ported | | | @@ -57,0 +66,6 @@ Token* ICUTokenizer::next(Token* token) { | Ported | | | @@ -64,0 +79,8 @@ Token* ICUTokenizer::next(Token* token) { | Ported | | | @@ -71,0 +94,5 @@ void ICUTokenizer::reset() { | Ported | | | @@ -75,0 +103,5 @@ void ICUTokenizer::reset() { | Ported | | | @@ -78,0 +111,30 @@ void ICUTokenizer::reset() { | Ported | | ### `be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/tokenizer/ik/ik_tokenizer_factory.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -0,0 +1,49 @@ | Adapted | IKTokenizerFactory and its smart/max-word construction are ported; only position_capability is omitted because the 4.1 TokenizerFactory interface has no such method. | ### `be/src/storage/index/inverted/tokenizer/keyword/keyword_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -45,0 +46,4 @@ public: | Ported | | | @@ -47,0 +52,3 @@ public: | Ported | | | @@ -71 +78 @@ private: | Ported | | ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -20,0 +21 @@ | Ported | | | @@ -76,0 +78,4 @@ Token* NGramTokenizer::next(Token* token) { | Ported | | | @@ -93,0 +99,6 @@ void NGramTokenizer::reset() { | Ported | | ### `be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -47 +47 @@ private: | Ported | | ### `be/src/storage/index/inverted/tokenizer/pinyin/pinyin_tokenizer.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/tokenizer/pinyin/pinyin_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/tokenizer/pinyin/smart_get_word.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/tokenizer/standard/standard_tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/src/storage/index/inverted/tokenizer/standard/standard_tokenizer_impl.h` | Source hunk | Status | Evidence | |---|---|---| | @@ -168,0 +169,4 @@ public: | Ported | | | @@ -176,0 +181 @@ public: | Ported | | | @@ -192,0 +198 @@ private: | Ported | | | @@ -293,0 +300 @@ private: | Ported | | | @@ -299 +306 @@ using StandardTokenizerImplPtr = std::unique_ptr<StandardTokenizerImpl>; | Ported | | ### `be/src/storage/index/inverted/tokenizer/tokenizer.h` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/exprs/function/function_search_nested_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/exprs/function/function_search_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -32,0 +33 @@ | Ported | | | @@ -187,0 +189,33 @@ private: | Adapted | SearchConvertsExceptionInsideSearchToStatus is adapted to the 4.1 V2 direct BKD reader; DummyInvertedIndexReader injects the same thrown exception. | | @@ -3887,0 +3922,39 @@ TEST_F(FunctionSearchTest, TestSearcherCacheHandlesLifetime) { | Adapted | SearchConvertsExceptionInsideSearchToStatus is adapted to the 4.1 V2 direct BKD reader; DummyInvertedIndexReader injects the same thrown exception. | ### `be/test/runtime/index_policy/index_policy_mgr_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -30,0 +31 @@ | Ported | | | @@ -34 +35,44 @@ namespace doris { | Ported | | | @@ -122 +166 @@ TEST_F(IndexPolicyMgrTest, TestApplyPolicyChanges) { | Ported | | | @@ -128 +172,52 @@ TEST_F(IndexPolicyMgrTest, TestApplyPolicyChanges) { | Ported | | | @@ -187,0 +283,212 @@ TEST_F(IndexPolicyMgrTest, TestTokenFilterProcessing) { | Ported | | | @@ -205 +512 @@ TEST_F(IndexPolicyMgrTest, AnalyzerProviderPreservesPurposeInsensitiveNormalizer | Ported | | ### `be/test/storage/index/inverted/analysis_factory_mgr_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/analyzer/ik_anayzer_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/char_filter/icu_normalizer_char_filter_factory_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp` → `be/test/storage/segment/inverted_index_reader_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -431,0 +432,25 @@ TEST_F(InvertedIndexReaderAnalysisPurposeTest, PartialAnalysisFailureDoesNotPubl | Adapted | ClassicReaderConvertsAnalyzerFailureToStatus is ported to be/test/storage/segment/inverted_index_reader_test.cpp; the source test file does not exist on 4.1. | ### `be/test/storage/index/inverted/similarity/collection_statistics_test.cpp` → `be/test/storage/compaction/collection_statistics_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -33,0 +34 @@ | Ported | | | @@ -37,0 +39 @@ | Ported | | | @@ -41,0 +44 @@ | Ported | | | @@ -46,0 +50 @@ | Adapted | The test moved to be/test/storage/compaction/collection_statistics_test.cpp; includes and fixture types follow the 4.1 layout. | | @@ -58,0 +63 @@ | Ported | | | @@ -138,0 +144,8 @@ private: | Ported | | | @@ -2087,0 +2101,22 @@ TEST_F(CollectionStatisticsTest, CollectUsesMatchRequestAnalyzerProvider) { | Adapted | The test moved to be/test/storage/compaction/collection_statistics_test.cpp; includes and fixture types follow the 4.1 layout. | | @@ -2181,0 +2217,63 @@ TEST_F(CollectionStatisticsTest, MatchSelectsOnlyTheRuntimeAnalyzerIndex) { | Adapted | The test moved to be/test/storage/compaction/collection_statistics_test.cpp; includes and fixture types follow the 4.1 layout. | | @@ -2600,0 +2699,30 @@ TEST_F(CollectionStatisticsTest, BuildFieldNameWithoutSuffix) { | Adapted | ScoringAnalyzerContextReportsWrongFamilyComponentAsStatus is ported using the 4.1 policy manager and analyzer_context_from_properties. | ### `be/test/storage/index/inverted/token_filter/icu_normalizer_filter_factory_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/token_filter/pinyin_filter_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/token_filter/word_delimiter_filter_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/tokenizer/icu_tokenizer_factory_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/tokenizer/keyword_analyzer_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `be/test/storage/index/inverted/tokenizer/ngram_tokenizer_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -61,0 +62,23 @@ TEST(NGramTokenizerTest, DefaultMinMaxValues) { | Ported | | | @@ -285,0 +309,32 @@ TEST(NGramTokenizerTest, WhitespaceTokenization) { | Ported | | ### `be/test/storage/index/inverted_index_parser_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -338 +338 @@ TEST_F(InvertedIndexParserTest, TestConstants) { | Ported | | | @@ -346,4 +346,4 @@ TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_EmptyInput) { | Ported | | | @@ -353,3 +353,3 @@ TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_MixedCase) { | Ported | | | @@ -358,2 +358 @@ TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_MixedCase) { | Ported | | | @@ -361 +360 @@ TEST_F(InvertedIndexParserTest, NormalizeAnalyzerKey_NoneParser) { | Ported | | | @@ -388 +387 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_CustomAnalyzerUpp | Ported | | | @@ -394 +393 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_Normalizer) { | Ported | | | @@ -400,0 +400,2 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_ParserKey) { | Ported | | | @@ -406,0 +408,2 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_ParserKeyAlias) { | Ported | | | @@ -430 +433 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_NormalizerOverrid | Ported | | | @@ -439 +442 @@ TEST_F(InvertedIndexParserTest, BuildAnalyzerKeyFromProperties_AnalyzerOverrides | Ported | | | @@ -462,0 +466,16 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_OnlyAnalyzerCustom) { | Ported | | | @@ -490 +509 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_BothAnalyzerAndParser) { | Ported | | | @@ -502 +521 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_AnalyzerNameOverridesParser | Ported | | | @@ -510 +529 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_AnalyzerNameOverridesParser | Ported | | | @@ -512,3 +531,4 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_CaseInsensitive) { | Ported | | | @@ -535,0 +556 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_AllBuiltinTypes) { | N-A | 4.1 has no PARSER_KUROMOJI enum or Kuromoji analyzer; the added built-in assertion has no target implementation. | | @@ -540,0 +562,2 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_AllBuiltinTypes) { | Ported | | | @@ -554,0 +578 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_IsBuiltinAnalyzer) { | N-A | 4.1 has no PARSER_KUROMOJI enum or Kuromoji analyzer; the added built-in assertion has no target implementation. | | @@ -555,0 +580,2 @@ TEST_F(InvertedIndexParserTest, AnalyzerConfigParser_IsBuiltinAnalyzer) { | Ported | | ### `be/test/storage/segment/inverted_index_iterator_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -22,0 +23 @@ | Ported | | | @@ -25,0 +27 @@ | Ported | | | @@ -29,0 +32,7 @@ | Ported | | | @@ -32,0 +42 @@ | Ported | | | @@ -54,0 +65 @@ public: | Ported | | | @@ -62,0 +74 @@ public: | Ported | | | @@ -106,0 +119,28 @@ protected: | Ported | | | @@ -116 +156 @@ TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_Uppercase) { | Ported | | | @@ -120 +160 @@ TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_MixedCase) { | Ported | | | @@ -124 +164 @@ TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_NonEmptyString) { | Ported | | | @@ -126 +166 @@ TEST_F(InvertedIndexIteratorTest, EnsureNormalizedKey_NonEmptyString) { | Ported | | | @@ -171,0 +212,408 @@ TEST_F(InvertedIndexIteratorTest, AddReader_MultipleReadersWithDifferentKeys) { | Ported | | | @@ -368 +816 @@ TEST_F(InvertedIndexIteratorTest, EdgeCase_EmptyAnalyzerKeyQuery) { | Ported | | | @@ -374,2 +822,5 @@ TEST_F(InvertedIndexIteratorTest, EdgeCase_CaseInsensitiveQuery) { | Ported | | | @@ -431,0 +883,77 @@ TEST_F(InvertedIndexIteratorTest, SelectBestReader_DeterministicByIndexId) { | Ported | | ### `be/test/storage/segment/inverted_index_writer_test.cpp` | Source hunk | Status | Evidence | |---|---|---| | @@ -35,0 +36 @@ | Ported | | | @@ -120,0 +122,28 @@ private: | Ported | | | @@ -1238,0 +1268,74 @@ TEST_F(InvertedIndexWriterTest, ErrorHandlingInFileWriter) { | Ported | | ### `fe/fe-catalog/src/main/java/org/apache/doris/analysis/AnalyzerSelector.java` → `fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzerSelector.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -24,0 +25 @@ import java.util.Collections; | Ported | | | @@ -110 +111 @@ public final class AnalyzerSelector { | Ported | | | @@ -114 +115 @@ public final class AnalyzerSelector { | Ported | | | @@ -121 +122 @@ public final class AnalyzerSelector { | Ported | | | @@ -148 +149 @@ public final class AnalyzerSelector { | Ported | | ### `fe/fe-catalog/src/main/java/org/apache/doris/analysis/MatchPredicate.java` → `fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -124,2 +124 @@ public class MatchPredicate extends Predicate { | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -3346,0 +3347,6 @@ public class SchemaChangeHandler extends AlterHandler { | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -27,0 +28,2 @@ import org.apache.doris.common.DdlException; | Ported | | | @@ -32,0 +35 @@ import com.google.common.base.Strings; | Ported | | | @@ -38,0 +42 @@ import java.util.List; | Ported | | | @@ -45,0 +50,6 @@ public class InvertedIndexUtil { | Ported | | | @@ -340 +350 @@ public class InvertedIndexUtil { | Ported | | | @@ -345,2 +355,2 @@ public class InvertedIndexUtil { | Ported | | | @@ -348,0 +359 @@ public class InvertedIndexUtil { | Ported | | | @@ -351,2 +361,0 @@ public class InvertedIndexUtil { | Adapted | The 4.1 method layout differs; exact policy spelling is resolved before parser-only normalization. | | @@ -356,0 +366,32 @@ public class InvertedIndexUtil { | Ported | | | @@ -395,0 +437 @@ public class InvertedIndexUtil { | Ported | | | @@ -397 +439,2 @@ public class InvertedIndexUtil { | Ported | | | @@ -401,0 +445,4 @@ public class InvertedIndexUtil { | Ported | | | @@ -405,0 +453,18 @@ public class InvertedIndexUtil { | Ported | | | @@ -425,0 +491,17 @@ public class InvertedIndexUtil { | Ported | | | @@ -428 +510 @@ public class InvertedIndexUtil { | Ported | | | @@ -433,2 +515,2 @@ public class InvertedIndexUtil { | Ported | | | @@ -436 +518,6 @@ public class InvertedIndexUtil { | Adapted | matchesBuiltinIkDefaults uses Guava ImmutableMap.of because the 4.1 Java source level does not provide Map.of. | ### `fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20 @@ package org.apache.doris.analysis.invertedindex; | Ported | | | @@ -24,0 +26,3 @@ import com.google.common.base.Strings; | Ported | | | @@ -26,0 +31,6 @@ import org.apache.logging.log4j.Logger; | Ported | | | @@ -27,0 +38 @@ import java.util.Map; | Ported | | | @@ -28,0 +40,2 @@ import java.util.TreeMap; | Ported | | | @@ -31,0 +45,28 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -35,0 +77,39 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -47,0 +128,10 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -49 +139,3 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -55 +147,113 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -73,3 +277,2 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -118,0 +322,24 @@ public final class AnalyzerIdentityBuilder { | Adapted | Guava ImmutableMap replaces Map.of for the 4.1 Java source level. | | @@ -126,0 +354,12 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -129 +368 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -133,0 +373 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -137 +377 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -139 +379 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -141 +381,4 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -143 +385,0 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -152,0 +395,6 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -157,4 +405,138 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -161,0 +544 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -163 +546,22 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -165,3 +569,5 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -168,0 +575,4 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -170,3 +580,9 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -174,2 +590,15 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -176,0 +606,21 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -178,3 +628,11 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -181,0 +640,7 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -183,6 +648,78 @@ public final class AnalyzerIdentityBuilder { | Adapted | Guava ImmutableList replaces List.of for the 4.1 Java source level. | | @@ -190,3 +726,0 @@ public final class AnalyzerIdentityBuilder { | Adapted | The builder was restructured around effective component identities; old fallback and string-building paths are removed or replaced in 4.1. | | @@ -193,0 +728,285 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -209,4 +1028,5 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -214,5 +1034,3 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -219,0 +1038,5 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -228,0 +1052,16 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -230 +1069 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -233 +1071,0 @@ public final class AnalyzerIdentityBuilder { | Adapted | The builder was restructured around effective component identities; old fallback and string-building paths are removed or replaced in 4.1. | | @@ -237,4 +1075,9 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -241,0 +1085,4 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -243,4 +1090,45 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -247,0 +1136,5 @@ public final class AnalyzerIdentityBuilder { | Ported | | | @@ -249 +1142,315 @@ public final class AnalyzerIdentityBuilder { | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerKeyNormalizer.java` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -4228,0 +4229 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc | Ported | | | @@ -4231,0 +4233,2 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc | Ported | | | @@ -4233,0 +4237,7 @@ public class OlapTable extends Table implements MTMVRelatedTableIf, GsonPostProc | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -62 +62,2 @@ public class IndexPolicy implements Writable, GsonPostProcessable { | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -43,0 +44 @@ import java.util.List; | Ported | | | @@ -57,0 +59,3 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -64 +68,50 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -82,0 +136,40 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -86,0 +180 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -95,6 +188,0 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Adapted | The old normalized-only lookup was replaced with exact-first helpers; reserved-name checks move after IF NOT EXISTS, and registerPolicyNameLocked maintains both name maps. | | @@ -103 +191,4 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -132 +223,5 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -139,10 +234,13 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -150,0 +249,3 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -153,4 +254,3 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -158,0 +259,12 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -159,0 +272 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -162 +275,4 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -168,0 +285,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -171,0 +295,7 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -184,13 +313,0 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Adapted | The old normalized-only lookup was replaced with exact-first helpers; reserved-name checks move after IF NOT EXISTS, and registerPolicyNameLocked maintains both name maps. | | @@ -198,0 +316,16 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -221,2 +353,0 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Adapted | The old normalized-only lookup was replaced with exact-first helpers; reserved-name checks move after IF NOT EXISTS, and registerPolicyNameLocked maintains both name maps. | | @@ -223,0 +355 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -234 +366 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -336,0 +469,11 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -349,12 +492 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -396,0 +529,4 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -470 +605,0 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -473 +608 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -485 +620 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -487 +622 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -496 +631 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -511 +646 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -514,2 +649 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -526,3 +660,2 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -545 +678 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -548,2 +681 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -560,3 +692,2 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -573 +703,0 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -588,2 +718,2 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -594 +724 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -597 +727 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -603,3 +733,13 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -608,2 +748,2 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -641,2 +781 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -659 +798 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | | @@ -680,7 +819,12 @@ public class IndexPolicyMgr implements Writable, GsonPostProcessable { | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -32,0 +33 @@ import org.apache.doris.analysis.FunctionParams; | Ported | | | @@ -258,0 +260,3 @@ public class ExpressionTranslator extends DefaultExpressionVisitor<Expr, PlanTra | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Match.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -51 +51 @@ public abstract class Match extends BinaryOperator implements PropagateNullable, | Ported | | | @@ -55 +54,0 @@ public abstract class Match extends BinaryOperator implements PropagateNullable, | Ported | | | @@ -58,2 +57 @@ public abstract class Match extends BinaryOperator implements PropagateNullable, | Ported | | ### `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateIndexOp.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -20,0 +21 @@ import org.apache.doris.alter.AlterOpType; | Ported | | | @@ -21,0 +23 @@ import org.apache.doris.catalog.Index; | Adapted | 4.1 uses org.apache.doris.analysis.IndexDef.IndexType; CreateIndexOp.validate still resolves policy names before translating the index. | | @@ -80,0 +83,4 @@ public class CreateIndexOp extends AlterTableOp { | Ported | | ### `fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -35,0 +36,4 @@ import org.apache.doris.common.jmockit.Deencapsulation; | Ported | | | @@ -1220 +1224 @@ public class SchemaChangeHandlerTest extends TestWithFeService { | Ported | | | @@ -1231,0 +1236,535 @@ public class SchemaChangeHandlerTest extends TestWithFeService { | Adapted | All 14 source-added test methods are present; Java 8 collection factories and the 4.1 catalog test fixture replace master-only APIs. | ### `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20 @@ package org.apache.doris.analysis; | Ported | | | @@ -20,0 +22,3 @@ import org.apache.doris.catalog.Env; | Ported | | | @@ -21,0 +26,2 @@ import org.apache.doris.catalog.PrimitiveType; | Adapted | All 27 source-added test methods are present in the new 4.1 test file; IndexType and Map factories use 4.1 APIs. | | @@ -22,0 +29 @@ import org.apache.doris.common.AnalysisException; | Ported | | | @@ -23,0 +31,6 @@ import org.apache.doris.indexpolicy.IndexPolicyMgr; | Ported | | | @@ -27,0 +41 @@ import org.junit.jupiter.api.Test; | Ported | | | @@ -30,0 +45 @@ import org.mockito.Mockito; | Ported | | | @@ -31,0 +47 @@ import java.util.HashMap; | Ported | | | @@ -35,0 +52,929 @@ public class InvertedIndexPropertiesTest { | Adapted | All 27 source-added test methods are present in the new 4.1 test file; IndexType and Map factories use 4.1 APIs. | | @@ -393,0 +1339,21 @@ public class InvertedIndexPropertiesTest { | Adapted | All 27 source-added test methods are present in the new 4.1 test file; IndexType and Map factories use 4.1 APIs. | ### `fe/fe-core/src/test/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilderTest.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -25,0 +26 @@ import org.apache.doris.indexpolicy.IndexPolicyTypeEnum; | Ported | | | @@ -30,0 +32 @@ import org.mockito.Mockito; | Ported | | | @@ -84 +86,4 @@ public class AnalyzerIdentityBuilderTest { | Ported | | | @@ -210,0 +216,2151 @@ public class AnalyzerIdentityBuilderTest { | Adapted | All 54 source-added behaviors are covered; max_ngram_diff-only assertions use supported default gram bounds because 4.1 lacks that property. | ### `fe/fe-core/src/test/java/org/apache/doris/indexpolicy/PolicyValidatorTests.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -19,0 +20 @@ package org.apache.doris.indexpolicy; | Ported | | | @@ -20,0 +22,3 @@ import org.apache.doris.catalog.Env; | Adapted | All 18 source-added test methods are present; invalid NGram replay uses the 4.1 min/max validator and Java 8 collections. | | @@ -21,0 +26 @@ import org.apache.doris.common.DdlException; | Ported | | | @@ -34,0 +40 @@ import java.io.DataOutputStream; | Ported | | | @@ -35,0 +42,2 @@ import java.util.HashMap; | Adapted | All 18 source-added test methods are present; invalid NGram replay uses the 4.1 min/max validator and Java 8 collections. | | @@ -72,0 +81,6 @@ public class PolicyValidatorTests { | Ported | | | @@ -264,0 +279,395 @@ public class PolicyValidatorTests { | Adapted | All 18 source-added test methods are present; invalid NGram replay uses the 4.1 min/max validator and Java 8 collections. | ### `fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/ExpressionTranslatorTest.java` | Source hunk | Status | Evidence | |---|---|---| | @@ -22,0 +23 @@ import org.apache.doris.analysis.Expr; | Adapted | The source-added MATCH translation test is present using the 4.1 expression and policy APIs. | | @@ -23,0 +25 @@ import org.apache.doris.analysis.IntLiteral; | Ported | | | @@ -24,0 +27 @@ import org.apache.doris.analysis.SlotRef; | Ported | | | @@ -26,0 +30,3 @@ import org.apache.doris.catalog.Type; | Ported | | | @@ -35,0 +42,2 @@ import org.apache.doris.nereids.types.IntegerType; | Ported | | | @@ -39,0 +48,4 @@ import org.junit.jupiter.api.Test; | Ported | | | @@ -59,0 +72,26 @@ public class ExpressionTranslatorTest { | Adapted | The source-added MATCH translation test is present using the 4.1 expression and policy APIs. | ### `regression-test/data/inverted_index_p0/analyzer/test_analyzer_identity_semantics.out` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `regression-test/data/inverted_index_p0/analyzer/test_ik_custom_analyzer.out` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `regression-test/suites/inverted_index_p0/analyzer/test_analyzer_identity_semantics.groovy` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `regression-test/suites/inverted_index_p0/analyzer/test_analyzer_malformed_utf8_write.groovy` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | ### `regression-test/suites/inverted_index_p0/analyzer/test_ik_custom_analyzer.groovy` | Source hunk | Status | Evidence | |---|---|---| | All hunks | Ported (verbatim) | | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
