airborne12 opened a new pull request, #68584: URL: https://github.com/apache/doris/pull/68584
### What problem does this PR solve? Issue Number: N/A (branch backport) Related PR: #66052, #66855, #66856, #66861, #66864, #66869, #66874, #66877, #66961, #67135, #67171, #67180, #67537, #67808 Problem Summary: Backport the final merged SNII implementation and its fixes from master to `branch-4.2`. Master and `branch-4.2` have different FE, BE, and cloud APIs, so this is a manual semantic backport. The source commits and original authorship are recorded in the first commit. The deployed SelectDB minimax 4.1.7 SNII writer uses `SniiStatsPB.null_count = 4` and `SniiSectionRefsPB.null_bitmap = 3`, with `bsbf = 4`. This backport preserves those field numbers, appends scoring fields at tag 5, and rejects removed CommonGrams metadata. The deployed writer did not emit CommonGrams metadata. Bundled segments from that writer are read by the default golden corpus test: 513 query observations, zero mismatches. The target branch already has #67859 through #68082 and the generic norms policy from #68039 through #68140. The SNII-specific norms policy and the ARM64 CRC32C guard from #67451 were adapted here. #67657 and the separate ngram/IK changes are outside this backport. ### Source PRs and hunk audit The [full audit table](https://github.com/airborne12/apache-doris/blob/5b052add5a4e0656437a325377135b4a010b3a7e/docs/backports/snii-branch-4.2-hunk-audit.tsv) records every source `@@` hunk, its target status, and the reason for adaptation or non-applicability. Its 3,188 rows include 2,962 ported, 174 adapted, and 52 non-applicable hunks. Of the 52 non-applicable hunks, 50 were removed by #67537, one refers to a build helper absent from `branch-4.2`, and one is an unused CommonGrams import. The 14 source commits were condensed into one manual backport commit, so a one-to-one `range-diff` is not meaningful; the per-hunk table provides the source-to-target comparison. | Source PR | Commit | Ported | Adapted | N/A | | --- | --- | ---: | ---: | ---: | | [#66052](https://github.com/apache/doris/pull/66052) | `96fd1e9bd2cf` | 935 | 124 | 27 | | [#66855](https://github.com/apache/doris/pull/66855) | `5870fb25134f` | 10 | 1 | 0 | | [#66856](https://github.com/apache/doris/pull/66856) | `72a587f54132` | 94 | 1 | 0 | | [#66861](https://github.com/apache/doris/pull/66861) | `80c1a3a27618` | 23 | 0 | 0 | | [#66864](https://github.com/apache/doris/pull/66864) | `1db4b1d3050f` | 13 | 0 | 0 | | [#66869](https://github.com/apache/doris/pull/66869) | `74b4418d738b` | 13 | 5 | 0 | | [#66874](https://github.com/apache/doris/pull/66874) | `511a8a0f8931` | 5 | 1 | 0 | | [#66877](https://github.com/apache/doris/pull/66877) | `40e1463f8c21` | 25 | 2 | 0 | | [#66961](https://github.com/apache/doris/pull/66961) | `e33e04823aa8` | 24 | 0 | 0 | | [#67135](https://github.com/apache/doris/pull/67135) | `fa95a70656fe` | 6 | 2 | 0 | | [#67171](https://github.com/apache/doris/pull/67171) | `40ff995f8908` | 13 | 1 | 0 | | [#67180](https://github.com/apache/doris/pull/67180) | `47b2e1d3d3a1` | 203 | 2 | 0 | | [#67537](https://github.com/apache/doris/pull/67537) | `9f3b084a7525` | 1,500 | 26 | 25 | | [#67808](https://github.com/apache/doris/pull/67808) | `59859162cda4` | 98 | 9 | 0 | ### Verification - FE build and Checkstyle passed; 43 focused FE unit tests passed. - BE and cloud builds passed. ASAN BE unit test runs passed with 1,334 SNII tests, 671 additional source tests, 87 target-adapted tests, and the default golden corpus test. Source-disabled performance benchmarks remain disabled. - All 13 related regression suites passed in an isolated cloud-mode FE/BE/MS environment: `test_storage_format_snii`, `test_storage_format_snii_custom_analyzer`, `test_storage_format_snii_norms`, `test_storage_format_snii_utf8_wildcard`, `test_multi_tokenize_index_not_built`, `test_phrase_candidate_pushdown`, `test_cloud_build_index_update`, `test_variant_search_subcolumn_snii`, `test_variant_v2_snii_index`, `regression_test_variant_snii_compaction`, `regression_test_variant_var_index_snii`, `test_variant_multi_index_nonCurrent`, and `test_variant_predefine_types_with_indexes_profile`. - Clang-format 16 and the merged-tree format gate passed. Clang analyzer passed for the SNII metadata, inverted parser, and segment iterator changes. Its full `column_reader.cpp` scan also reports two unchanged baseline initialization warnings in `FileColumnIterator` and `PageReadOptions`; their source lines are identical to `branch-4.2`. - The English description check passed for 520 source files. Its 22 reviewed literal exceptions are Unicode query, writer, or regression input data; no comments, diagnostics, or identifiers are exempted. ### Release note Backport SNII inverted index storage format support to `branch-4.2`. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test (deployed-format readback described above) - Behavior changed: - [x] Yes. SNII storage format support is available on this branch. - Does this need documentation? - [x] No. This PR backports the merged source feature without adding new syntax. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
