yiguolei opened a new pull request, #70479: URL: https://github.com/apache/doris/pull/70479
### Background MoW DELETE statements can be rewritten as INSERT SELECT operations that write a delete sign. ANN indexes are built when segments are written, including for delete-sign rows. Group Commit converts incoming rows and queues them for a shared background load; async mode can acknowledge the request after writing its WAL. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: DELETE on a MoW table with an ANN index can fail because the rewritten write omits the vector column, which is then filled with an empty array. Group Commit can acknowledge a malformed vector before the segment writer rejects its dimension, causing the shared background load and subsequent WAL replay to fail. Preserve indexed vectors in DELETE projections and validate ANN dimensions in the Group Commit sink before buffering or enqueueing the converted rows. ### Details - **Change map:** Approximately 96 lines of production code and 279 lines of tests. The FE change adds ANN columns to the DELETE projection; the BE change resolves existing ANN metadata and validates retained rows before enqueue/WAL. The remaining changes add FE and BE unit tests and an integration regression suite. - **Frequency:** Deterministic for delete-sign rewrites that omit ANN vectors and for retained Group Commit rows whose array length differs from the ANN dimension. - **Root cause:** Missing vectors in delete-sign writes are filled with empty arrays. Generic Group Commit array validation does not enforce the ANN dimension, leaving that failure to the background segment writer. - **Why existing tests missed it:** The existing ANN MoW regression supplies vectors explicitly when inserting delete signs. It does not exercise SQL DELETE rewrites or assert immediate Group Commit rejection of malformed vectors. - **Fix:** Carry every ANN-indexed column from the DELETE query into its sink. Resolve ANN column positions from existing schema metadata during Group Commit prepare, then check each retained row after conversion and filtering and before buffering/enqueueing. Resolve materialized-index column IDs through names rather than treating them as output slot positions. - **Other paths and branches:** The shared DELETE rewrite also covers DELETE USING. The Group Commit sink covers stream loads and INSERT in sync and async modes. General partial updates, light-delete predicates, WAL replay, and ANN row-ID mapping are unchanged. No release-branch backport is included. - **Behavior change:** Retained vectors with incorrect dimensions fail the Group Commit request even when max_filter_ratio permits ordinary row filtering. Tombstones still participate in ANN construction with their original vectors. This does not change async durability guarantees or provide request-wide rollback for valid batches already enqueued before a later batch fails. - **Compatibility and cost:** No Thrift, protobuf, storage-format, or configuration changes. FE already sends ANN definitions through the shared OlapTableSink schema builder. Validation scans array offsets once per ANN column and retained batch; DELETE additionally reads and writes indexed vectors. ### Testing Passed locally: - build-support/check-build-hygiene.sh - clang-format-11 --dry-run --Werror on the changed BE source, header, and unit test - git diff --check Added, but not executed locally: - DeleteFromCommandTest: preserve multiple ANN vector columns while leaving ordinary columns omitted; preserve the existing key-only rewrite for tables without ANN. - GroupCommitAnnValidationTest: valid and invalid row dimensions across calls, empty batches, ordinary arrays, multiple ANN columns, constant/nullable wrappers, metadata resolution, and invalid metadata. - ann_delete_group_commit: key and value DELETE predicates, DELETE USING, sync/async stream loads with short/empty/overlong vectors, multiple indexed columns, mixed valid/invalid rows, async INSERT rejection, and a subsequent valid load. The BE UT entry point was attempted but exited during environment initialization because this macOS host has no JDK. FE build prerequisites also lack thirdparty/installed/bin/protoc. Compilation, unit tests, and the regression suite therefore remain unverified and need CI or a configured Doris development environment. The regression uses inline result assertions; no golden output files were fabricated without executing the suite. ### Release note Fix DELETE failures on MoW tables with ANN indexes caused by missing vector values. Group Commit now rejects retained vectors with incorrect ANN dimensions before writing them to the queue or WAL, rather than reporting async success and later failing the background load. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: static checks passed; compilation, unit tests, and integration regression execution are pending. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Production changes, tests, and this PR description. - Human review: The author discussed the fix scope and metadata flow; line-by-line review and runtime validation remain pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
