yiguolei opened a new pull request, #70552: URL: https://github.com/apache/doris/pull/70552
### Background ANN indexes store vector entries using segment row IDs. Empty arrays can occur in delete rows, but the current writer rejects them as dimension mismatches; simply omitting them would shift HNSW labels and make range-search complements include rows without vectors. Group Commit also needs to reject invalid dimensions before accepting blocks into its background write path. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Add an explicit opt-in for empty arrays in ANN-indexed columns while preserving data-file row IDs. Nonempty vectors retain native IVF labels, and HNSW uses IDMap only when a segment contains empty arrays. Approximate distance functions return NULL for empty arrays, including scalar fallback, and sparse-segment TopN uses normal SQL evaluation and sorting to preserve NULL ordering and LIMIT. Existing writes continue to reject empty arrays by default. ### Details - **Change map:** 524 changed production lines and 961 changed test lines; no generated files. The production changes cover writer validation/row IDs and native FAISS persistence, range-search domains, Group Commit validation, and nullable approximate-distance evaluation/materialization. Tests cover writer state, FAISS reload/filtering, reader contracts, expression nullability, and SQL differential checks. - **Before and after:** With the flag disabled, an empty ANN array fails with an actionable error. With it enabled, empty rows remain in the data file but have no ANN entry. Range predicates exclude them on both sides of a comparison; scalar approximate distance returns NULL. Sparse TopN falls back so NULLS FIRST/LAST and limits larger than the vector count remain correct. - **Configs and defaults:** New mutable BE config `enable_ann_index_empty_array=false`. After writing empty arrays, keep it enabled, including across restarts, compaction, index rebuilds, and Group Commit WAL replay. Readers do not consult the flag. - **Impact on existing users:** Dense segments retain the existing native index representation and TopN path. Dense IVF complements do not scan all list IDs. Legacy NULL-coordinate writer behavior is preserved so this change does not reject historical rows during rewrites; scalar distance NULL-input errors remain unchanged. - **Behavior change:** `l2_distance_approximate` and `inner_product_approximate` are nullable in FE and return NULL if either array is empty in scalar evaluation. Nonempty dimension mismatches remain errors. Exact distance functions are unchanged. Sparse-segment TopN trades ANN acceleration for correct SQL NULL handling. - **Compatibility:** Upgrade BEs first, then FEs; enable empty-array writes only after every FE and BE is upgraded. The new BE preserves the non-nullable output layout of plans from old FEs. Once empty arrays have been written, downgrade to a BE without this support is unsupported: native FAISS deserialization alone does not make old complement, HNSW range-search, empty-IVF, or rewrite paths compatible. Disabling the flag does not convert persisted data. New FEs with old BEs are not a supported mixed-version configuration for this change; rolling back before opt-in requires reverting FEs before BEs and cross-version verification. - **Implementation:** All-empty segments persist an empty index without training; complement results retain the required null distance/row-ID pointers. Sparse range domains are derived lazily from FAISS IDs, protected by a mutex, and published only after successful initialization. HNSW range queries translate selectors and returned labels around the child index to preserve derived search parameters with the pinned FAISS version. The FAISS dependency and on-disk filenames are unchanged. ### Testing Passed locally on macOS arm64: - Repository clang-format v16 scripts (`clang-format.sh` and `check-format.sh`) on a scoped snapshot of the 22 modified C++ files. - `bash build-support/check-build-hygiene.sh`. - `git diff --cached --check`. Added/extended tests: - Writer opt-in rejection without partial buffer mutation, cross-batch sparse labels, all-empty finish, and legacy NULL-coordinate writes. - HNSW/IVF/IVF_ON_DISK native reload, selector filtering, both range directions, concurrent domain initialization and IO failure/retry, and the dense-IVF fast path. - Reader-level all-empty range contracts for both metrics and every index type. - Approximate empty-array NULLs, constant query vectors, old-FE result layout, nonempty dimension errors, and unchanged exact-distance behavior; FE nullable expressions. - Group Commit validation and regression coverage for off/sync/async writes. Differential SQL checks compare forced scalar fallback with index execution and cached results, including sparse/all-empty segments and NULL ordering. No handwritten result files. Attempted but blocked before tests ran: - `bash run-be-ut.sh --run --filter='AnnIndexWriterTest.*:AnnIndexReaderTest.*:VectorSearchTest.*:GroupCommitAnnValidationTest.*:FunctionArrayApproximateDistanceTest.*:ExtractQueryVectorTest.*' -j4`. - `bash run-fe-ut.sh --run org.apache.doris.nereids.trees.expressions.functions.scalar.ApproximateDistanceTest`. - `bash run-regression-test.sh -d ann_index_p0 -s ann_empty_array_queries,ann_delete_group_commit`. The environment has no Java/JDK 17 or installed third-party build dependencies. BE/FE compilation, Checkstyle, clang-tidy, and runtime tests have not completed. Compaction/restart/WAL recovery and actual old/new-binary upgrade/downgrade scenarios still need validation. This PR is a draft pending those checks. ### Release note Add opt-in support for empty arrays in ANN-indexed columns through `enable_ann_index_empty_array`, disabled by default. Approximate distances return NULL for empty arrays; sparse segments use scalar TopN evaluation to preserve NULL ordering. Upgrade all FEs and BEs before enabling the flag, and keep it enabled after writing such data. Downgrading to a BE without sparse ANN support after these writes is unsupported. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document the BE config, NULL semantics, upgrade order, and downgrade restriction in the ANN user documentation before marking ready. - Pre-submission status: Not ready: draft pending compilation, runtime tests, recovery tests, and cross-version validation. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Code, tests, and this description. - Human review: Design and compatibility constraints discussed with the author; final implementation review pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
