yiguolei opened a new pull request, #70454: URL: https://github.com/apache/doris/pull/70454
### Background Faiss ANN index training and vector insertion share a BE-wide OpenMP thread budget controlled by `omp_threads_limit`. Each call currently reserves half of the remaining budget, so concurrent calls can use different OpenMP thread counts. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Make the OpenMP reservation for each ANN train/add call independent of concurrent budget usage. Add `ann_index_build_threads_per_task`, cap its effective value by the total budget, and wait until a full reservation is available. This preserves aggregate admission control while making per-call parallelism predictable. Most added lines are unit tests. ### Details - **Before and after:** Previously, a total budget of 10 could produce reservations of 5, 2, and 1 as calls arrived. With 4 threads per task, two calls reserve 4 each and a third waits until 4 slots are free. Faiss work executes outside the budget mutex; RAII releases reservations and wakes waiters on completion or failure. - **Configs and defaults:** Add non-dynamic BE configuration `ann_index_build_threads_per_task`, default `-1`: use 1 thread below 32 available CPU cores and 4 otherwise. Positive explicit values are accepted, zero and values below -1 are rejected. Cap the reservation by `omp_threads_limit`. Keep the existing total-budget default of 80% of available cores, at least 1. Both auto defaults use one CPU snapshot captured on first use; a nonpositive core count logs a warning and falls back to 16. Configuration changes require a BE restart. - **Impact on existing users:** Default per-call parallelism changes, which can change build latency and concurrency. The total configured ANN budget is retained. A remainder smaller than one reservation stays unused until another call finishes. This controls reserved OpenMP parallelism, not a hard OS CPU quota. - **Behavior change:** Yes; fixed full reservations replace half-of-remaining reservations for both ANN training and vector insertion. Other branches are unchanged. ### Testing - Passed locally: `build-support/check-format.sh` with clang-format 16.0.5, `build-support/check-build-hygiene.sh`, and `git diff --check`. - Added unit tests for configuration validation, explicit and automatic reservations, capping to the total budget, stable CPU snapshots, concurrent train/add admission, and reservation release after a training exception. - Attempted `./run-be-ut.sh --run --filter='AnnOmpThreadBudgetTest.*:VectorSearchTest.OmpThreadBudgetNeverExceedsLimit' -j 4` on macOS arm64. Environment initialization stopped at the Java version check before compilation; the tests have not run. BE compilation and clang-tidy have not run. ### Notes for Reviewers This change was motivated by a persistent OpenBLAS GEMM hang with 12 logical jobs and six observed workers executing only even-numbered jobs. Varying caller thread counts interacting with shared OpenBLAS thread-count state is a source-supported hypothesis, not a reproduced root cause. This PR removes varying ANN build reservations; it does not claim to fix every possible OpenBLAS deadlock. The new tests exercise admission and reservation behavior, not the native OpenBLAS race. ### Release note Add the BE configuration `ann_index_build_threads_per_task` to control threads reserved per ANN train/add call while retaining `omp_threads_limit` as the total budget. The automatic per-call default is 1 thread below 32 available CPU cores and 4 otherwise; calls wait when a full reservation is unavailable. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document the new BE configuration and its defaults in apache/doris-website as a follow-up. - Pre-submission status: Partially verified: formatting and build hygiene passed; compilation, unit tests, and clang-tidy remain unverified. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests, and this description. - Human review: The author reviewed and directed the budget design, defaults, CPU snapshot behavior, and invalid-core fallback; full code review is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
