shyjsarah opened a new pull request, #73:
URL: https://github.com/apache/paimon-vector-index/pull/73
## Summary
This is a follow-up to #72. After #72 removes completed queries from later
automatic IVF batch rounds, incomplete queries still rerun the full larger
`nprobe` and rescan old lists. This PR makes those retries incremental: a `16
-> 32 -> 64` expansion scans `[16,32)` and then `[32,64)` instead of rescanning
`[0,32)` and `[0,64)`.
## Changes
- pass half-open probe ranges through automatic batch expansion for
IVF-FLAT, IVF-SQ, IVF-PQ, and IVF-RQ
- seed each expansion round with the query's accumulated Top-K so prior heap
results are retained
- scan only newly added lists in filtered and unfiltered batch paths
- add optional aggregate diagnostics controlled by
`PAIMON_VINDEX_LOG_IVF_PROGRESSIVE_SEARCH`
- one line per expansion round
- one completion summary
- no vectors or row IDs are logged
- keep fixed-width/scalar searches, storage formats, public APIs, and the C
ABI unchanged
## Testing
- `cargo fmt --check`
- `cargo test -p paimon-vindex-core`
- 450 passed, 1 ignored
- `cargo clippy -p paimon-vindex-core --all-targets -- -D warnings`
- added coverage for:
- half-open probe ranges across multiple expansion rounds
- carrying accumulated Top-K seeds between rounds
- preserving a valid row ID of `-1` via its finite distance
- IVF-PQ range partitioning and seeded result equivalence with a one-shot
search
- aggregate diagnostic formatting
## Notes
- Depends on #72; the second commit is the PR2-specific change.
- This eliminates repeated inverted-list payload reads and scans. Query
preprocessing and coarse centroid selection are still recomputed for the
smaller active-query set in each round; no cross-round public search-state API
is introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
