JunRuiLee opened a new issue, #755: URL: https://github.com/apache/paimon-rust/issues/755
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation paimon-rust can already run a primary-key (bucket-local ANN) vector search end to end on its own — that was #514, now closed. What it cannot do is act as the **execution kernel for an engine that plans the query somewhere else**. That is the shape needed to bring PK-vector search to external Paimon tables in a distributed engine. Apache Doris is being wired to read Paimon through the paimon-rust C FFI (apache/doris#65883 adds a `PaimonRustReader` on the BE side). For a vector search over an external Paimon table, planning happens in Paimon **Java** on the Doris FE, and each unit of work is shipped to a BE, which calls into paimon-rust. Java already produces the right unit of work. `PrimaryKeyVectorScan` aggregates a whole bucket into a single `BucketVectorSearchSplit` (`BucketAccumulator.build()`), so a bucket is never split across tasks and the ANN current-segment decision stays correct. apache/paimon#9386 then gave that class a public, versioned byte form (`PKVSPLIT` v1), so a non-Java engine can receive one. What is missing on the Rust side is everything after decoding those bytes: - plan **from** the split — its payload files, its per-file row ranges, and the snapshot it pins — instead of re-enumerating the index manifest for the whole table; - run search, optional refine, Top-K and materialization for that split; - expose it over the C FFI so a BE can call it. Worth doing inside paimon-rust rather than in each engine: the semantics here are Paimon's, not the engine's — exact fallback over the data files no ANN segment covers, deletion-vector filtering of stale candidates, a deterministic global Top-K, and materialization by physical row position. Reimplementing that per engine is how it drifts. ### Solution Five steps. The first two are already open as PRs. **1. Decode the `BucketVectorSearchSplit` byte form — #746 (open).** `BucketVectorSearchSplit::deserialize` for the `PKVSPLIT` v1 envelope (`i64` magic + `i32` version + embedded `DataSplit.serialize` + `IndexFileMetaSerializer.serializeList` + per-file row ranges), with golden fixtures produced by Java. One thing to settle at or before step 5: `deserialize_binary_array_str` validates only that each variable-length region is within bounds — it does not check element ordering, aliasing, or trailing bytes, so `n` elements may all point at the same large body and each get cloned, giving an output bound around `len²/8`. It is reachable from `DataFileMeta` row decoding in `crates/paimon/src/spec/data_file.rs`, therefore from the embedded `DataSplit`. #746 already enforces the writer's real invariants for the row-array variant (each element body starts exactly at the previous padded end; the array ends exactly at the last element's padded end); the same rule should be applied to the string variant before a C entry point starts accepting arbitrary split bytes. **2. Resolve index files by external path and bucket layout — #752 (open, draft).** `PkVectorScan` currently builds every index path as `<table>/index/<file>` and ignores both `external_path` and bucket-local placement (`index-file-in-data-file-dir`). Java can write either. Split-driven execution has to land after this, or it will fail to find index files that Java legitimately placed elsewhere. **3. Extract the search seams in `vector_search_builder` (not yet opened).** Behaviour-preserving refactor of `plan_and_search_pk_candidates_batch` into (a) query/parameter resolution and (b) search over an externally supplied plan — exposing **both** the raw indexed/exact candidate layer and the merged/reranked layer. The raw layer is what a candidate-only phase would need later; extracting only the merged layer would mean reopening this refactor then. No behaviour change, no new public API. **4. Plan from a decoded bucket split (not yet opened).** `PkVectorScan::plan_for_bucket_vector_splits`: build the plan from the split's payload files and `rowRangesByFile` rather than reading the index manifest; require that all supplied splits pin the same snapshot; apply partition pruning; and keep `PkVectorScanPlan.snapshot_id` populated, since it stays authoritative even when pruning leaves zero searchable splits. One asymmetry to handle explicitly: Java only inserts a `rowRangesByFile` entry for `IndexedSplit` files, while the Rust kernel treats a missing key as "no rows allowed". Files omitted from the map must be normalized to unrestricted full-file ranges, or valid splits will silently return nothing. **5. Execute a bucket split end to end, plus the C entry point (not yet opened).** `VectorSearchBuilder::execute_read_for_bucket_splits` and `paimon_vector_search_builder_execute_read_for_bucket_splits`: split bytes in; search, optional refine, local Top-K and row materialization in one call; Arrow record batches with `__paimon_search_score` out. Query parameters and projection reuse the existing `with_*` builder methods. Covered by an end-to-end test driven by Java-produced split bytes. After step 5 an engine can distribute one call per bucket and merge the per-bucket Top-K itself. ### Anything else? **A staged form is deliberately left out of the five steps above.** Java + Spark does something stronger: search tasks return lightweight candidates, the driver merges them globally, reranks only the survivors, and emits `IndexedSplit`s that executors read back. With the per-bucket form above, refine and full-row reads happen within a bucket, so a multi-bucket query can rerank and materialize rows that the global Top-K then discards. Adding candidate-only search, global merge/rerank, and deferred materialization is the natural follow-up — but the candidate and materialize wire formats should not be frozen before there is a consumer for them, so it is better proposed separately. **Related gaps, independent of this lane.** `DataSplit` decoding accepts only versions 8 and 9 while Java reads 1..9 (the current Java writer always emits 9). `DataSplit.total_buckets` is stored as `i32` with an absent value silently read as `1`, while Java models it as nullable — normal planning always sets it and PK-vector tables require fixed or postpone bucketing, so this is a wire-conformance gap rather than a blocker for the steps above. **No new Java API is required.** `BatchVectorSearchBuilder` (partition filter, filter, limit, vector column, vectors, `newVectorScan()`), `VectorScan.scan()`, `PrimaryKeyVectorScan.Plan.splits()`/`snapshotId()` and `BucketVectorSearchSplit.serialize(DataOutputView)` are all public as of apache/paimon#9386. Predecessor: #514 (primary-key vector read, closed). Prerequisite already merged: #745 (accept `DataSplit` version 9). ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
