JunRuiLee opened a new pull request, #771:
URL: https://github.com/apache/paimon-rust/pull/771
## Why
Final step of #755. paimon-rust can already run a primary-key vector search
that it plans itself,
but not act as the execution kernel for an engine that plans elsewhere —
which is the shape Doris
needs for an external Paimon table (apache/doris#65883): planning runs in
Paimon Java on the FE,
and each bucket is shipped to a BE that calls in here.
Steps 1-3 built up to it — decode the split bytes (#746), resolve index
files by external path and
bucket layout (#752), plan from a decoded split (#757) — but
`plan_for_bucket_vector_splits` had no
caller, so none of it was reachable. This adds the part that runs the search
and returns rows, and
the C symbol a BE calls.
## How
`VectorSearchBuilder::execute_read_for_bucket_splits` takes the splits as
the plan: their payload
files, per-file row ranges and pinned snapshot are used as given, and the
index manifest is not
read. Everything after planning is the ordinary primary-key vector read —
search, optional refine,
Top-K, materialization are shared with `execute_read`, not reimplemented —
so the output is the
same: projected user columns plus `__paimon_search_score`, best-first. The
Top-K is local to the
splits passed in; a caller distributing one call per bucket merges the
per-bucket results itself.
Rejected rather than worked around:
- **A data-evolution column.** It plans through the global index, so
answering it would silently use
a different plan than the caller supplied.
- **An empty split list.** It pins no snapshot, and would otherwise be
indistinguishable from a
query that matched nothing.
`paimon_vector_search_builder_execute_read_for_bucket_splits` exposes this
over the C ABI as a NEW
symbol (the existing terminal's signature is pinned by the guard block). The
array-of-buffers
marshalling is the only logic the C layer adds: a null array, a zero count,
or a null entry is the
caller's input error and is reported as one, rather than reaching the
decoder as corrupt data.
Also drops the two `#[allow(dead_code)]` attributes #757 added for the entry
point, now that it has
a caller.
## The fixture, and what makes the test able to fail
Both halves come from ONE Java run: the table directory written by the
`ivf-flat` indexer, and the
split bytes its own `PrimaryKeyVectorScan` planned over that same table. A
split names its files by
generated UUID, so bytes captured separately would reference files that do
not exist. Generated by
a new `PkVectorSplitFixtureGenerator` in `paimon-vector` — I'll send that to
apache/paimon
separately; the command and config are recorded in the test's header comment
so the fixture can be
regenerated.
Worth flagging for review: **"the split route agrees with the manifest
route" is not a sufficient
test.** It is also what a read that quietly ignored the split and re-planned
from the manifest would
produce — I checked, by substituting the manifest plan: all of the obvious
assertions still passed.
What separates them is the split's per-file row ranges, which nothing else
carries. The fixture's
own `rangeFileCount` is zero — not an oversight, but the asymmetry #755
called out: Java records a
range only for a file its pre-filter narrowed, and this query has no
pre-filter. So the committed
bytes exercise the omitted-file normalization (omitted = whole file, against
a kernel that reads a
missing key as "no rows"), and `restricts_the_read_to_the_splits_row_ranges`
builds the listed case
on top. With the manifest plan substituted, that test is the one that fails.
The split embeds its bucket directory as an absolute path, because that is
what Java serializes and
what a real engine ships, so the test stages the fixture into a temp dir and
rewrites that one
`writeUTF` string.
## Testing
6 integration tests over the Java fixture, 2 C-binding tests for the
marshalling.
Gates: `cargo fmt --all -- --check`; `cargo clippy -p paimon --all-targets
-- -D warnings` and
`-p paimon-c`, both clean; `cargo test -p paimon --lib` 2586 passed,
`--features fulltext` 2661
passed, `--tests` all green.
`cargo test -p paimon-c` is 72 passed / 1 failed —
`vector_search_append_filter_returns_invalid_input`
fails identically on unmodified `main` (70 passed / 1 failed, same test), so
it is pre-existing and
not from this change. Happy to look at it separately.
## Not here
The staged Spark-style form (candidate-only search, driver-side global
merge/rerank, deferred
materialization) — #755 leaves it out deliberately: with the per-bucket
form, refine and full-row
reads happen within a bucket, so a multi-bucket query can materialize rows
the global Top-K then
discards. The candidate wire format shouldn't be frozen before there is a
consumer.
An unrestricted split still materializes a whole-file allow-list where Java
returns no mask at all,
which costs Lumina its batch-search path. #757 noted this belongs with the
entry point that makes
the route reachable, but saying "no mask" changes the plan's own type, so
I'd rather do it as its
own change than widen this one — say the word if you'd prefer it folded in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
