goankur opened a new pull request, #16656:
URL: https://github.com/apache/lucene/pull/16656
### Title:
Parallel `O_DIRECT` full-precision rerank reads for larger-than-RAM KNN
search
### Description
Two-phase KNN — quantized (BBQ) graph search in RAM, then full-precision fp32
rerank of the shortlist — keeps recall high while shrinking the resident
footprint. But the rerank reads the `fp32` vectors serially, and once the
index
exceeds RAM those reads hit disk, where serializing a query's
few-hundred-vector
shortlist dominates p99.
This PR:
- **`ParallelVectorReadable`** (core, `store`): an optional `IndexInput`
capability to fetch a batch of fixed-size float vectors at scattered
offsets,
possibly in parallel.
- **`RescoreTopNQuery` / `FullPrecisionFloatVectorSimilarityValuesSource`**:
the
full-precision rescorer reads the shortlist via that capability, off the
main
field's own raw fp32 through a new `Lucene99FlatVectorsReader` accessor (no
duplicate rerank field).
- **`Lucene99FlatVectorsWriter`**: 4 KB page-aligns FLOAT32 vector data so
each
read is a single block (padding is transparent to readers).
- **`SelectiveDirectIODirectory`** (misc): opens only `.vec` with `O_DIRECT`
and
implements the capability; it fetches the shortlist through a
**caller-owned
read `Executor`**, sized independently of the searcher's executor. The core
read path is Directory-agnostic; mmap inputs fall back to serial reads.
### Benchmarks
25M Cohere-v3 (1024-dim), 102 GB index, process capped to 10 GB (~10× smaller
than the fp32 working set) so rerank reads hit disk. Local NVMe, fio 136k
random-4KB IOPS / 1.12 GB/s. Single-stream, oversample 5, fanout 100.
| Metric | mmap baseline | O_DIRECT rerank (this PR) |
|---|---|---|
| p99 latency | 176 ms | 23.5 ms |
| recall | 0.963 | 0.963 |
| read IOPS | 35.4k | 50.9k |
| avg read size | ~28 KB (readahead-amplified) | 4 KB |
Under concurrent load the design meets p99 ≤ 50 ms at ~240–245 QPS on this
box
(one-off validation numbers, not committed).
### Notes for reviewers
- Per-query rerank read concurrency is a **storage concern**: it follows the
caller-owned read `Executor` on `SelectiveDirectIODirectory`, decoupled
from
the searcher's executor so a larger-than-RAM deployment can keep enough
reads
in flight to saturate the device without widening CPU-side search
parallelism.
- The capability interface is float-vector-specific (mirrors
`IndexInput.readFloats`) rather than a general byte-oriented batch read — a
deliberate choice to avoid a decode copy on the rerank hot path. Happy to
generalize if preferred.
- New test `TestSelectiveDirectIODirectory` covers parallel + serial reads,
shuffled offsets, and the empty shortlist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
