JingsongLi opened a new pull request, #822: URL: https://github.com/apache/paimon-rust/pull/822
### Purpose Vector search planning, DE/PK execution, and row materialization were concentrated in `vector_search_builder.rs`, with separate result and execution paths for primary-key tables. This change exposes a shared Scan → Plan → Read flow, following the Java API structure, so native and externally planned searches use the same readers. Related: #771. ### Brief change log - Separate the single-query and batch builders, DE/PK scanners and readers, query validation, result reads, and their tests into focused modules. - Introduce common `VectorScan`, `VectorScanPlan`, `VectorRead`, and `BatchVectorRead` APIs. Readers own their configuration, and plans can be reused across queries without replanning. - Unify DE global row IDs and PK physical positions under a snapshot-scoped `SearchResult`, with projected row materialization through `new_read_builder()`. Update Hybrid and DataFusion callers. - Move Java bucket-split byte decoding to a standalone C API. Construct a common plan from decoded split handles and execute it through the shared C reader, preserving supplied snapshots, files, and row ranges. ### Tests - Core vector unit tests: 451 passed (`cargo test -p paimon --offline --features fulltext --lib vector`). - PK baseline, batch, bucket-split, and Java-fixture integration suites: 26 passed. - Hybrid search builder tests: 12 passed. - C vector-search tests: 15 passed. - DataFusion vector-search tests: 7 passed. - Generated the C header and compiled the documented reader example with C11, `-Wall -Wextra -Werror`; compiled the Java serializer example with `javac`. - `git diff --check` passed. Regression coverage includes plan reuse across single and batch readers, handle lifetimes, plan/reader context mismatches, invalid C inputs, and reading Java-planned PK splits after removing the index manifest. ### API and Format This intentionally changes Rust vector-search APIs: builders return the common `SearchResult`, and row reads are configured from that result. The bucket-specific Rust execution entry and C `paimon_vector_search_builder_execute_read_for_bucket_splits` entry are removed in favor of separate decoding, plan construction, and reading. The ordinary C `paimon_vector_search_builder_execute_read` remains a local convenience operation. No persisted table or split format changes. The standalone decoder accepts Java's versioned `PKVSPLIT` format; Java object-stream envelopes and serialized DE index/raw splits are not supported. DE uses the common native scan/plan/read API. ### Documentation Extend `docs/src/c-binding.md` with the shared vector API and a complete Java-to-C example covering split serialization, table metadata, scalar residuals, Arrow ownership, cleanup, plan reuse, and distributed Top-K merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
