jeho-rpls opened a new pull request, #16368: URL: https://github.com/apache/lucene/pull/16368
Fixes #16367. HNSW graph construction never checked `OneMerge`'s abort flag — it is a pure-CPU loop with no writes, so `IndexWriter#rollback` / `abortMerges()` blocked until the entire graph was built (tens of minutes for production-sized vector segments; measurements in the issue). ### Changes - `HnswBuilder` gains `setAbortCheck(IORunnable)`; `HnswGraphBuilder` invokes it before every node insertion in `addGraphNodeInternal` — the single point all insertion paths funnel through (full rebuild via `addVectors`, graph-join via `MergingHnswGraphBuilder`, and `HnswConcurrentMergeBuilder` workers, which forward the check). - `Lucene99HnswVectorsWriter` passes `mergeState::checkAborted` into the graph mergers (new constructor overloads; existing constructors unchanged). - Flush-time graph builds are unaffected (no check set), matching the null contract of `checkIntegrity(OneMerge)` from #16281. - Overhead: one null check per node insertion, where each insertion costs hundreds of vector comparisons. ### Testing - New `TestHnswMergeAbort` exercises both merge paths end-to-end (rollback during a `forceMerge`): full rebuild (every segment >40% deleted, so no base graph is eligible) and graph join (deletion-free segments). Without the fix both fail — rollback blocks 26 s (rebuild) / 31 s (join) on a 48k x 96d index; with the fix it returns in milliseconds. - `./gradlew tidy` and `:lucene:core:check` pass. - The approach was also validated on a production Elasticsearch 9.4.2 cluster via a 10.4 backport: node shutdown during a live HNSW merge went from 24–31 min (grace period exceeded, force-killed) to 2.2 s, and undisturbed merges complete unchanged (details in #16367). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
