raghavyadav01 opened a new pull request, #20072: URL: https://github.com/apache/pinot/pull/20072
Adds an opt-in parallel build to `HnswVectorIndexCreator`. HNSW graph construction runs on one thread today, which dominates segment build time for vector columns. With the vector index property `buildThreads` > 1, vectors are buffered (copied, since callers reuse the array). On `seal()`, they are indexed from that many threads: each thread writes a contiguous doc-id range into its own `IndexWriter`, and the parts are attached with `addIndexes`. New vector index properties (all optional; the defaults keep today's behavior): | Property | Default | Meaning | |---|---|---| | `buildThreads` | `1` | threads used to build the graph on seal | | `forceMerge` | `true` | merge into one Lucene segment on seal; `false` keeps the per-thread graphs as separate segments, searched together | | `parallelBuildMaxDocs` | `50000` | above this many vectors, buffering stops and the creator falls back to the single writer, so large segments keep the existing memory profile | Doc ids are stored per document as before, so every mode maps results back to the right row. **Tests**: `HnswVectorIndexCreatorTest#testParallelBuildKeepsDocIds` builds 1,000 vectors with the default writer, 4 threads with and without force-merge, and a case that outgrows the buffer. In each, it checks that searching for a stored vector finds its row and that the per-thread directories are removed. The existing HNSW creator, combined-index, handler and rollback tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
