suxiaogang223 opened a new pull request, #70704:
URL: https://github.com/apache/doris/pull/70704

   ### Background
   
   Paimon primary-key vector indexes can select nearest neighbors before 
returning rows to Doris. The branch-4.2 implementation in #69493 adds SQL ANN 
pushdown and a `vector_search` TVF; master now plans external scans through 
connector plugins.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #69493 (source implementation), #64304 (connector migration), 
#70075 (reserved protocol), #70598 (Rust 0.5.0 dependency)
   PR Type: Other
   
   Problem Summary:
   
   Move the reviewed Paimon ANN implementation and additional local tests onto 
master without replacing its newer connector architecture or downgrading its 
SDKs. This Draft currently contains the native reader and test migration; the 
FE connector, SQL rewrite and TVF entry points still need adaptation. It is not 
ready to merge and does not yet enable ANN queries on master.
   
   ### Details
   
   - **Change map:** Production code: +758/-47 lines in three reader/converter 
files. Tests and SDK-generated fixture metadata: +1810/-0 lines in 61 files, 
including binary fixtures; no generated regression outputs, documentation or 
dependency changes. The native reader opens FE-planned bucket vector splits, 
validates requests and converts scores; tests exercise exact materialization, 
row ranges, NULLs, metrics, cleanup and malformed input.
   - **Why Other:** This is an in-progress migration Draft, rather than a 
complete feature submission. Its type and description will be updated when the 
connector and SQL entry points are connected.
   - **Behavior change:** The BE can consume reserved Paimon vector payloads. 
The master FE does not emit them yet.
   - **Compatibility:** Reuse master's Java SDK 2.1, paimon-rust 0.5.0 and 
reserved Thrift fields. Preserve master's adaptive batch slicing, predicate 
profile counters and normal-read behavior. Mixed-version query errors are 
acceptable; avoiding process crashes remains required. Rust eager 
materialization memory budgeting remains deferred.
   - **Remaining migration:** Add a connector SPI contract for vector-search 
admission/planning, move Paimon eligibility and split generation into its 
plugin, adapt SQL TopN/slot propagation and TVF binding, then migrate the 
corresponding FE tests. Keep master SDK/MERGE fixes rather than replay 
branch-specific compatibility backports.
   
   ### Testing
   
   - Local `build-support/check-build-hygiene.sh`: passed all header layering, 
closure, instantiation and unity checks.
   - Local clang-format 16 formatting and format check: passed.
   - Local `git diff --cached --check`: passed.
   - The seven new native fixture tests passed ASAN/UBSAN on the source branch 
before this migration; they have not been compiled or run against this master 
candidate or Rust 0.5.0.
   - Full FE/BE build and external Spark/Doris regression: not run. The ANN 
regression suite is carried forward for completion of the FE migration and 
cannot pass its ANN entry points yet.
   
   ### Notes for Reviewers
   
   Keep this PR in Draft while the FE migration is in progress. Native fixtures 
use Java Paimon 2.1-generated compacted Parquet files with a FULL search plan; 
they do not validate an actual ANN index. The source branch and its original 
uncommitted local tests are preserved.
   
   ### Release note
   
   None; this migration Draft is not ready for release.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (document SQL ANN and `vector_search` 
once the FE interface migration is complete)
   - Pre-submission status: Not ready: FE connector admission, SQL pushdown and 
TVF migration are incomplete; master compilation and runtime validation are 
pending.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6)
   - Generated: Migration edits, additional native/regression tests and this 
description; source ANN implementation originates in #69493.
   - Human review: Not stated for the master migration.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to