Authorlove opened a new pull request, #69313: URL: https://github.com/apache/doris/pull/69313
### What problem does this PR solve? Issue Number: None Related PR: https://github.com/apache/doris-thirdparty/pull/420 Problem Summary: Native CLucene compaction copied source token totals even when it removed documents, so an index built by compaction could score and rank the same visible rows differently from a freshly built index. Update the CLucene dependency to accumulate exact frequencies of retained postings per destination field. Norms-bearing indexes with `support_phrase=false` do not retain the frequencies needed to reconstruct exact token totals. Select the existing raw-column rebuild path for their columns instead of invoking native CLucene compaction. The selection examines every sibling index because CLucene compaction eligibility is currently tracked per column. Normal frequency-bearing indexes retain native compaction; SNII selection is unchanged. The regression compares scores and TopK/offset results of independently built clean indexes with full-compacted histories across V2/V3, `support_phrase=true/false`, delete-sign rows and overwritten versions. It does not encode a new BM25 formula or require immediate visible-row statistics before compaction. This PR depends on the linked third-party change. Keep it in draft until that dependency's final commit is available and the Doris integration checks pass. ### Release note Correct BM25 token statistics after CLucene inverted-index compaction removes or redistributes documents. Norms-bearing indexes without stored frequencies are rebuilt from column data during compaction. ### Check List (For Author) - Test - [x] Unit Test (dependency: 16 ASAN tests passed; new tests fail on its original implementation) - [x] Manual test (pre-fix reproduction and native-merge versus rebuild comparison on Doris 4.1.5) - [ ] Regression test (added; local framework build stopped during test dependency download, so the new SQL suite has not run) - [ ] Full Doris master BE/FE build and end-to-end execution - `build-support/clang-format.sh` and `build-support/check-format.sh` with clang-format 16. - `build-support/check-build-hygiene.sh` and `git diff --check`. - Full Doris build/clang-tidy requires the current third-party build bundle; no `compile_commands.json` is available yet. - Behavior changed: - [x] Yes. Corrected compaction statistics can change scores/ranking. Norms-bearing CLucene indexes without frequencies use raw-column rebuilds during compaction. - Does this need documentation? - [x] No. No new configuration or storage format. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
