yiguolei opened a new pull request, #70788: URL: https://github.com/apache/doris/pull/70788
### Background ANN indexes accelerate vector TopN and range predicates. A segment without the requested ANN index can still be evaluated by scanning its vector data. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: ANN queries currently turn every returned index-loading error into brute-force fallback, hiding failures such as I/O and memory errors. Restrict fallback to a missing index and propagate other loading errors to the query caller. After an upgrade, queries that previously hid these failures report errors instead. Most added lines cover the missing-index and failure paths in unit tests. ### Details - **Before and after:** Previously, the boolean loading API treated every non-OK status as a reason to fall back. A single Status-returning `load_index` API now lets TopN and range callers fall back only for `INVERTED_INDEX_FILE_NOT_FOUND`; other statuses propagate. Missing required files inside an existing ANN directory remain errors. - **Configs and defaults:** None. The behavior applies whenever these ANN loading paths execute, without a new switch. - **Impact on existing users:** Queries may now fail on index-loading errors that were previously hidden by fallback. Missing-index fallback, small-candidate selection, persisted index formats and query syntax are unchanged. - **Behavior change:** Non-missing index-loading failures are reported to callers instead of silently selecting brute force. ### Testing Ran on a Linux development host with ASAN. The final commit has the same Git tree as the tested snapshot. ```bash ./run-be-ut.sh --run --filter='*AnnIndexLoadTest*:*AnnIndexReaderTest*:*AnnIndexIteratorTest*:VectorSearchTest.*' -j 16 ``` - All 124 tests from 5 suites passed, including 34 new `AnnIndexLoadTest` cases. - New cases cover init/open status propagation, repeated failed loads, TopN/range fallback versus errors, null-reader rejection, and missing internal `ann.faiss` files in HNSW, IVF and IVF_ON_DISK indexes. - Regression suites were not run. ### Notes for Reviewers - Normalize `INVERTED_INDEX_SNII_NOT_FOUND` only at the logical-index open lookup. Missing internal Faiss files must not be mistaken for an absent index. - Removing the `BE_TEST` loading shortcut lets tests exercise the real loader. Existing search-only mocks explicitly model successful loading instead. ### Release note ANN queries continue to use brute-force evaluation when a segment's ANN index is absent, but now report other index-loading failures instead of silently falling back. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes (plan: document missing-index-only fallback and propagation of other loading errors in apache/doris-website) - Pre-submission status: Partially verified: the selected ANN BE unit tests passed; end-to-end regression suites were not run. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests and this description. - Human review: Not stated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
