goankur opened a new pull request, #16666:
URL: https://github.com/apache/lucene/pull/16666
### Title:
(Implemented with AI)
Sandbox `IoUringDirectory`: batched `O_DIRECT` rerank reads via io_uring
### Depends on #16656
This builds on #16656 and will not compile without it: `UringInput implements
org.apache.lucene.store.ParallelVectorReadable`, the capability interface
added there. #16656
supplies the search-path wiring (`FlatVectorsReader.readRawVectors` → the
codec's
`ParallelVectorReadable` check); this PR only adds a second storage backend
behind that same hook,
so the two can be reviewed independently. **Nothing outside
`lucene/sandbox/` is touched.**
### Description
In #16656 a rerank shortlist is fetched by fanning ~350–500 `pread` calls
across a read thread
pool. That works, but each read costs a syscall, a thread hand-off, and a
blocked worker. io_uring
replaces the whole pattern: one thread prepares the entire shortlist as SQEs
and issues a single
`io_uring_enter` (submit-and-wait) against a pre-registered `O_DIRECT`
buffer, then reaps
completions in a batch. No per-read syscall, no cross-thread dispatch, and
no read-thread pool at
all.
- **`IoUringDirectory`** (`FilterDirectory`): serves only `.vec` through
io_uring; every other file
goes to the wrapped `FSDirectory` and stays page-cached. Opt-in via
constructor or
`-Dlucene.store.ioUring=true`, and a **transparent pass-through** when a
recent kernel or
`liburing-ffi` is missing — io_uring is never a hard dependency of a
running Lucene.
- **`IoUring`** (package-private engine): pure-FFM binding of `liburing-ffi`
— no JNI and no
third-party Java wrapper; liburing owns the ring mmap, memory ordering and
arch specifics. Rings
come from a **bounded pool** (sized to read concurrency, not to how many
threads ever reranked),
and the `O_DIRECT` fd is owned by the `IndexInput`, so a merged-away
segment's reader releases the
file instead of pinning it for the directory's lifetime.
### Why this was worth doing: the bottleneck was neither CPU nor the device
The pread path was measured at its own SLA point (130 QPS Poisson) and the
machine looked *idle*:
| Signal | Measured | Reading |
|---|---|---|
| `mpstat` | 31% busy, **50% idle**, iowait 18.7% | not CPU-bound |
| PSI `cpu.pressure` | `some` 5% | threads rarely queue for a core |
| PSI `io.pressure` | `some` ~24%, **`full` ~19%** | for ~19% of wall time
*no* runnable task could run — all waiting on I/O |
| eBPF `offcputime --state 2` | **82.8 thread-seconds blocked in a 12 s
window** (~6.9 threads parked), 81% in the read pool | stack: `pread64 →
xfs_file_dio_read → iomap_dio_rw → io_schedule_timeout` |
| `iostat` vs `fio` ceiling | ~47k of 136k IOPS | device had ~3× headroom |
So the loss was *waiting*, and faster storage would not have fixed it. Worth
recording because an
on-CPU profile says the opposite: `cpu-clock` attributed **79.8%** of
samples to stage-1 scoring and
only **11.4%** to the read path, while a **wall-clock** profile of the same
run attributed **3.6%**
and **78.1%** respectively. cpu-clock cannot see a blocked thread, so it
hides exactly the time that
sets p99. `perf stat` then showed the plumbing directly: **188k
context-switches/s against 91k
`pread64`/s ≈ 2.3 thread swaps per 4 KB read**, pure dispatch overhead.
### Benchmark setup
- **Box**: AWS g6.4xlarge — 16 vCPU AMD EPYC 7R13 (Zen 3), 60 GB RAM, local
NVMe instance store
measured with `fio` at **136k random-4KB read IOPS / 1.1 GB/s**.
- **Data**: 25M Cohere-v3 Wikipedia embeddings, 1024-dim fp32; 1-bit BBQ +
fp32 rerank; HNSW
maxConn 64 / beamWidth 250; **102 GB index**.
- **Larger-than-RAM**: searcher pinned to a **10 GB cgroup v2 hard limit**
(swap off, 4 GB heap), so
the graph and quantized codes stay resident and every rerank read hits the
SSD — ~10× the cap.
- **Queries**: 10,000, recall measured against exact top-100 ground truth
(GPU brute force) on every
configuration, so no result here trades accuracy for speed. Page cache
dropped before each run.
- **Single-stream**: run to completion, per-query latency.
- **Concurrent**: open-loop with **Poisson arrivals**, non-blocking
dispatch, a **bounded** handler
pool (32 threads = 2× cores, 32-deep queue) and **load shedding** rather
than unbounded queueing;
latency measured from *intended* arrival time (coordinated-omission-safe);
120 s per rate.
**SLA-QPS is the highest rate holding p99 ≤ 50 ms with ≤ 0.1% shed.**
### Results
Single-stream, oversample 5 / fanout 100, 4 KB-aligned index — same recall,
same read count, only
the submission mechanism differs:
| Metric | pread pool (#16656) | io_uring (this PR) |
|---|---|---|
| p99 latency | 18.3 ms | **14.1 ms** (−23%) |
| 10k queries wall-clock | 148 s | **110 s** |
| recall | 0.967 | 0.967 |
| avg queue depth | 3.8 | **7.6** |
| read IOPS / avg read size | 35.1k / 4 KB | 42.9k / 4 KB |
Concurrent, oversample 3.5 / fanout 25 (the recall-tuned operating point):
| Metric | pread pool | io_uring |
|---|---|---|
| **SLA-QPS** (p99 ≤ 50 ms, shed ≤ 0.1%) | ~130 | **~250 (+92%)** |
| p99 @ 240 QPS (fixed-rate diagnostic) | 46.8 ms | **19.7 ms** |
| context-switches @ 240 QPS | ~188k/s | **~33k/s** |
| `pread64` syscalls @ 240 QPS | ~91k/s | ~0 (`io_uring_enter` ~7k/s) |
| CPU used @ 240 QPS (`perf stat`) | 10.3 of 16 cores | **8.2 of 16 cores** |
| recall | 0.937 | 0.937 |
io_uring holds p99 ≤ 50 ms through 250 QPS (36.2 ms @190, 42.7 @230, 48.6
@250) with **zero
shedding at every rate**, failing only at 270 (62.9 ms) — while the pread
path crosses 50 ms by
~150 QPS and sheds 14% at 170. It is also *cheaper*: better latency on ~2
fewer cores.
### Testing
`TestIoUringDirectory` extends **`BaseDirectoryTestCase`**, so the full
`Directory`/clone/slice
contract runs against this directory (2 tests skip because a
`FilterDirectory` is not an
`FSDirectory` subclass — the same two that upstream `TestDirectIODirectory`
skips). On top of that,
8 targeted tests: randomized batch equivalence (random sizes, repeated and
unordered positions,
batches larger than the queue depth), unaligned positions, agreement with
`MMapDirectory`,
concurrency over a deliberately undersized 2-ring pool, reopen-after-close,
disabled pass-through,
merge-context exclusion, and dims of 1/512/1025/2048/3000 to exercise
multi-block spans.
Two invariants that only fail in rare conditions are asserted through
package-private counters and
were **mutation-verified** — each defect was reinstated to confirm the test
fails, then reverted:
a batch that aborts must destroy its ring rather than return it to the pool
(`ringCount`), and
closing an input must release its descriptor (`openFdCount`). Verified on
**x86_64 and aarch64**
(different `O_DIRECT` flag values); `./gradlew tidy check` is clean.
### Notes for reviewers
- **Runtime native dependency.** This needs `liburing-ffi.so.2` present to
do anything, which is
unlike anything else Lucene ships. It is opt-in, absent-safe, and
quarantined in sandbox — but
whether Lucene wants it *at all*, versus an external module, is a fair
question and I have no
stake in the answer.
- **Why not `NativeAccess`?** That SPI is hint-oriented
(`madvise`/`fadvise`) rather than a general
downcall framework, so io_uring does not fit it. The FFM style here
deliberately mirrors
`PosixNativeAccess`: `MH$name` handles, `invokeExact` in a try/catch that
throws `AssertionError`,
`UnsupportedOperationException` for a missing symbol.
- **O_DIRECT is verified, not assumed.** `open(2)` may ignore an
unrecognised `O_DIRECT` flag on an
unfamiliar architecture and silently give buffered reads, so `openDirect`
confirms with
`fcntl(F_GETFL)` and falls back to the delegate if the kernel did not
honour it.
- **Alignment handling.** O_DIRECT requires block-aligned offset, length
*and* buffer, so each
vector is fetched as the aligned span containing it and copied out at its
offset within that span.
Unaligned vector data therefore costs one extra block per vector — which
is why #16656's 4 KB
alignment halves the bytes read rather than merely tidying the layout.
- **Memory.** Each ring registers `queueDepth × 4 KB`; the default 256/64
can hold ~64 MB of
registered buffers off-heap. Size `maxRings` to the read concurrency you
want.
- **Known gaps** (why this is sandbox, not core): no deterministic test
forces the
partially-reaped-submission case that the ring-discard logic guards;
NRT/merge behaviour under
sustained churn is untested at scale; there is no minimum kernel/liburing
version policy yet; and
only the `.vec` file is served this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
