msokolov commented on code in PR #12660:
URL: https://github.com/apache/lucene/pull/12660#discussion_r1367971257
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene95/Lucene95HnswVectorsWriter.java:
##########
@@ -635,17 +667,31 @@ private static DocsWithFieldSet writeVectorData(
return docsWithField;
}
+ private HnswGraphMerger createGraphMerger(
+ FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier) {
+ if (exec != null) {
+ return new ConcurrentHnswMerger(
+ fieldInfo, scorerSupplier, M, beamWidth, exec, numMergeWorkers);
+ }
+ return new IncrementalHnswGraphMerger(fieldInfo, scorerSupplier, M,
beamWidth);
+ }
+
@Override
public void close() throws IOException {
IOUtils.close(meta, vectorData, vectorIndex);
+ if (exec != null) {
+ exec.shutdownNow();
+ }
+ System.out.println(
+ "Total contention time: " + NeighborArray.contentionTime.get() /
1000000 + " ms");
Review Comment:
Oooh, that's quite a bit! Almost 1/3 of the time spent waiting for locks! I
wonder how it would be w/2 threads or 4, and then again with 1M docs. In any
case the speedup is great and we can still run the CPU on other tasks while
we're waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
