Mryange opened a new pull request, #70394: URL: https://github.com/apache/doris/pull/70394
### Background Each segment builds an in-memory Faiss index before serializing it. ADD INDEX keeps the segment column writers alive until the entire rowset is closed. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Finished ANN writers retain their Faiss indexes, so build memory accumulates across segments during ADD INDEX. Release the index and input buffer when each build finishes, including failed, skipped, and aborted builds. Also end the swapped-out input buffer's lifetime before serialization. ### Details - **Frequency:** ADD INDEX over rowsets with multiple ANN segments. - **Root cause:** The writer retains its Faiss shared_ptr after serialization until rowset teardown. - **Why existing tests missed it:** They checked build results without checking destruction while the writer remains alive. - **Fix:** Share cleanup across skip/error paths and defer it in `_build_and_save()`. - **Other paths and branches:** Ordinary writes and compaction use the same ANN writer; file formats and close ordering are preserved. 4.1 has a cherry-pick conflict in `ann_index_writer.cpp`; 4.2 applies cleanly but uses the old array-offset API, incompatible with the new test. Neither pick label is requested. ### Testing - Passed clang-format 16, build hygiene, and `git diff --check` on the development machine. - Added seven parameterized cases checking index destruction and buffer release on success, train/add/save failure, empty/undersized segments, and abort. The success case also checks that the input buffer is freed before save. - Builds and runtime unit/regression tests were not run; the development-machine workflow requires separate build authorization. ### Release note Reduce retained memory during ANN index construction by releasing each segment's Faiss build index immediately after completion. ### Check List (For Author) - Behavior changed: Yes (build-resource lifetime only; see Details) - Does this need documentation: No - Pre-submission status: Partially verified: builds and runtime tests have not been run. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Code, tests, and PR description. - Human review: Implementation approach discussed with the author; final diff review pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
