JingsongLi opened a new pull request, #130: URL: https://github.com/apache/paimon-vector-index/pull/130
DiskANN queries can hang when a storage adapter uses `ThreadPool::install` on an independent Rayon I/O pool: the waiting query worker can reenter another query that waits for the suspended loader to publish its cache entry. Bound shared-cache waiting with one 10 ms deadline per read batch, then batch-read missing windows into query-local buffers without taking over the original loader's publication or cancellation rights. IVF-FLAT and IVF-SQ now reuse the metadata allocation protections already used by IVF-PQ/RQ: validate the estimated serialized-plus-decoded peak before allocating, probe the end of large metadata sections, and use fallible allocation. Failed FLAT loads retain no decoded arrays, and optional cache allocation failures disable retention. Regression coverage includes process-timeout tests for both adjacency and vector cache reentry, multi-window fallback batching and I/O failure/retry, truncated metadata, extreme headers, and FLAT shape revalidation. Update the cache behavior documentation and benchmark comments. Validation: - `cargo test --workspace --quiet` — 767 passed, 2 ignored. - `cargo build --all-targets` - `cargo clippy --all-targets --workspace -- -D warnings` - `cargo fmt --all -- --check` - `python3 tools/check_license_headers.py` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
