JunRuiLee opened a new pull request, #647:
URL: https://github.com/apache/paimon-rust/pull/647
### Purpose
This is PR 3/3 of a stacked vector-search performance series.
Depends on:
1. #645
2. #646
PK-vector bucket search previously did not execute all independent search
work
concurrently. ANN segments and exact-fallback files were scheduled in
separate
phases, and ANN CPU work could run inline or serially despite a configured
`global-index.thread-num` greater than one.
This change represents ANN segments and uncovered exact files as independent
leaves, executes them through one bounded concurrency window, and moves
blocking
index searches to a shared process-level executor.
While the preceding PRs are open, GitHub's main-based diff also contains PR 1
and PR 2. The isolated change in this PR is commit `80367d7`.
### Brief change log
- Combine ANN-segment and exact-file work into one concurrent bucket leaf
plan.
- Load each ANN segment lazily inside its leaf instead of preloading every
index.
- Keep vindex segments range-backed and Lumina segments buffered.
- Run blocking ANN and regular global vector-index searches on a dedicated
process-level executor.
- Apply one per-query concurrency budget across buckets, ANN segments, and
exact
fallback files.
- Apply a shared process-level budget across concurrent queries.
- Drain all submitted leaves and report errors deterministically by
submission
order.
- Preserve deterministic Top-K merging regardless of leaf completion order.
- Validate `global-index.thread-num` against Tokio's semaphore limit.
### Tests
- `cargo check -p paimon`
- `cargo test -p paimon vindex::executor --lib`
- `cargo test -p paimon vindex::pkvector --lib`
- `cargo test -p paimon table::pk_vector_orchestrator --lib`
- `cargo test -p paimon table::vector_search_builder --lib`
- `cargo test -p paimon test_global_index_thread_num --lib`
- `cargo test -p paimon --test pk_vector_baseline_test`
- `cargo test -p paimon --test pk_vector_batch_test`
- `cargo test -p paimon --test pk_vector_java_fixture_test`
- `cargo clippy -p paimon --lib --tests -- -D warnings`
- `cargo fmt --all -- --check`
- `git diff --check`
### API and Format
No public API or storage-format changes.
The concurrency configuration retains the existing
`global-index.thread-num` option and default value.
### Documentation
No user-facing documentation changes are required. The concurrency and
execution
changes are internal to vector-index reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
