suxiaogang223 opened a new pull request, #70704: URL: https://github.com/apache/doris/pull/70704
### Background Paimon primary-key vector indexes can select nearest neighbors before returning rows to Doris. The branch-4.2 implementation in #69493 adds SQL ANN pushdown and a `vector_search` TVF; master now plans external scans through connector plugins. ### What problem does this PR solve? Issue Number: None Related PR: #69493 (source implementation), #64304 (connector migration), #70075 (reserved protocol), #70598 (Rust 0.5.0 dependency) PR Type: Other Problem Summary: Move the reviewed Paimon ANN implementation and additional local tests onto master without replacing its newer connector architecture or downgrading its SDKs. This Draft currently contains the native reader and test migration; the FE connector, SQL rewrite and TVF entry points still need adaptation. It is not ready to merge and does not yet enable ANN queries on master. ### Details - **Change map:** Production code: +758/-47 lines in three reader/converter files. Tests and SDK-generated fixture metadata: +1810/-0 lines in 61 files, including binary fixtures; no generated regression outputs, documentation or dependency changes. The native reader opens FE-planned bucket vector splits, validates requests and converts scores; tests exercise exact materialization, row ranges, NULLs, metrics, cleanup and malformed input. - **Why Other:** This is an in-progress migration Draft, rather than a complete feature submission. Its type and description will be updated when the connector and SQL entry points are connected. - **Behavior change:** The BE can consume reserved Paimon vector payloads. The master FE does not emit them yet. - **Compatibility:** Reuse master's Java SDK 2.1, paimon-rust 0.5.0 and reserved Thrift fields. Preserve master's adaptive batch slicing, predicate profile counters and normal-read behavior. Mixed-version query errors are acceptable; avoiding process crashes remains required. Rust eager materialization memory budgeting remains deferred. - **Remaining migration:** Add a connector SPI contract for vector-search admission/planning, move Paimon eligibility and split generation into its plugin, adapt SQL TopN/slot propagation and TVF binding, then migrate the corresponding FE tests. Keep master SDK/MERGE fixes rather than replay branch-specific compatibility backports. ### Testing - Local `build-support/check-build-hygiene.sh`: passed all header layering, closure, instantiation and unity checks. - Local clang-format 16 formatting and format check: passed. - Local `git diff --cached --check`: passed. - The seven new native fixture tests passed ASAN/UBSAN on the source branch before this migration; they have not been compiled or run against this master candidate or Rust 0.5.0. - Full FE/BE build and external Spark/Doris regression: not run. The ANN regression suite is carried forward for completion of the FE migration and cannot pass its ANN entry points yet. ### Notes for Reviewers Keep this PR in Draft while the FE migration is in progress. Native fixtures use Java Paimon 2.1-generated compacted Parquet files with a FULL search plan; they do not validate an actual ANN index. The source branch and its original uncommitted local tests are preserved. ### Release note None; this migration Draft is not ready for release. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes (document SQL ANN and `vector_search` once the FE interface migration is complete) - Pre-submission status: Not ready: FE connector admission, SQL pushdown and TVF migration are incomplete; master compilation and runtime validation are pending. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Migration edits, additional native/regression tests and this description; source ANN implementation originates in #69493. - Human review: Not stated for the master migration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
