JunRuiLee commented on code in PR #771:
URL: https://github.com/apache/paimon-rust/pull/771#discussion_r3956756381
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -416,6 +439,160 @@ impl<'a> VectorSearchBuilder<'a> {
.await
}
+ /// Search bucket splits an engine planned elsewhere, and return WHAT the
search
+ /// found -- not the rows.
+ ///
+ /// Step one of the two-step primary-key vector read, mirroring Java
+ /// `PrimaryKeyVectorRead.read(plan)`. Reading the rows is step two and
belongs
+ /// to the caller's own read: hand these splits to
+ ///
[`TableRead::to_arrow_indexed`](crate::table::TableRead::to_arrow_indexed)
+ /// with whatever projection the caller wants.
+ ///
+ /// One [`PkVectorIndexedSplit`] per data file the search selected rows
from, in
+ /// ascending `(partition, bucket, file)` order, each carrying that file's
+ /// selected physical positions and their scores. Same shape as Java's
+ /// `IndexedSplit`, which is what its `PrimaryKeyVectorResult` hands to
the scan.
+ ///
+ /// The unit of work is Java's `BucketVectorSearchSplit` byte form: a
planner
+ /// running in Paimon Java enumerates one split per bucket -- a bucket is
never
+ /// divided, because the ANN current-segment decision needs the bucket's
whole
+ /// active file set -- and ships each to a worker that calls this. The
splits
+ /// are the plan: their payload files, their per-file row ranges and the
+ /// snapshot they pin are used as given, and this table's index manifest
is not
+ /// read.
+ ///
+ /// Search, optional refine and local Top-K all happen here, as they do in
Java's
+ /// `createResult`. The Top-K is local to the supplied splits: a caller
distributing
+ /// one call per bucket does its own global merge. It can read [`scores`]
here to
+ /// decide, without materializing anything, whether a bucket is worth
reading at
+ /// all -- but a split cannot be trimmed (construction is crate-private),
so the
+ /// splits it does read come back whole and the final Top-K happens on the
rows,
+ /// over `__paimon_search_score`. Java merges its candidates before
building
+ /// splits; a caller here cannot, because it holds splits rather than
candidates.
+ ///
+ /// [`scores`]: PkVectorIndexedSplit::scores
+ ///
+ /// This builder's [`with_filter`](Self::with_filter) still applies: it is
the
+ /// pre-Top-K scalar residual and must run before the search. Its
+ /// [`with_projection`](Self::with_projection) does NOT -- projection
belongs to
+ /// the read.
+ ///
+ /// The splits are NOT in best-first order and the read does not reorder
them:
+ /// rows come back in physical order carrying `__paimon_search_score`, and
a
+ /// caller wanting them ranked sorts on that column. Java does the same --
its
+ /// read emits physical order and `VectorSearchProcedure` sorts afterwards.
+ ///
+ /// Only a primary-key vector column can be searched this way. The
+ /// data-evolution route plans through the global index rather than through
+ /// bucket splits, so it is rejected rather than silently answered from a
+ /// different plan.
+ pub async fn search_for_bucket_splits(
Review Comment:
Thanks. I removed `search_for_bucket_splits` from `VectorSearchBuilder` and
moved the execution to `VectorRead`.
The API is now:
```rust
let vector_read = builder.new_vector_read()?;
let indexed_splits = vector_read.read(bucket_splits).await?;
let rows = table_read.to_arrow_indexed(&indexed_splits)?;
The externally planned BucketVectorSearchSplits are used directly without
re-planning from the manifest. The C API only decodes the split bytes and calls
the same VectorRead implementation.
```
I deliberately did not introduce a public VectorScan::Plan or
new_vector_scan() in this PR, because this integration already receives the
externally planned bucket splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
