Pulkitg64 commented on code in PR #16627:
URL: https://github.com/apache/lucene/pull/16627#discussion_r4149577845
##########
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java:
##########
@@ -90,6 +113,23 @@ public OnHeapHnswGraph build(int maxOrd) throws IOException
{
worker.setMergeStartTimeNs(mergeStartTimeNs);
worker.setCumulativeWorkTimeNs(cumulativeWorkTimeNs);
}
+ if (prunedGraph != null) {
+ long repairStartNs = System.nanoTime();
+ repairDisconnectedNodes();
+ long rebalanceStartNs = System.nanoTime();
+ prunedGraph.builder().rebalanceGraph();
+ long rebalanceEndNs = System.nanoTime();
+ if (infoStream.isEnabled(HNSW_COMPONENT)) {
+ infoStream.message(
+ HNSW_COMPONENT,
+ String.format(
+ Locale.ROOT,
Review Comment:
Non blocking: Should we add num nodes repaired too?
##########
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java:
##########
@@ -42,10 +43,13 @@ public class HnswConcurrentMergeBuilder implements
HnswBuilder {
private static final int DEFAULT_BATCH_SIZE =
2048; // number of vectors the worker handles sequentially at one batch
+ // Number of disconnected nodes a repair worker claims atomically at a time.
+ private static final int REPAIR_BATCH_SIZE = 64;
Review Comment:
Is there any reason behind this size? I mean why 64?
##########
lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java:
##########
@@ -90,6 +113,23 @@ public OnHeapHnswGraph build(int maxOrd) throws IOException
{
worker.setMergeStartTimeNs(mergeStartTimeNs);
worker.setCumulativeWorkTimeNs(cumulativeWorkTimeNs);
}
+ if (prunedGraph != null) {
+ long repairStartNs = System.nanoTime();
+ repairDisconnectedNodes();
+ long rebalanceStartNs = System.nanoTime();
+ prunedGraph.builder().rebalanceGraph();
Review Comment:
Just out of curiosity, did you explore making rebalancing concurrent too
i.e. once we have list of nodes which we need to add to a given level, we add
them concurrently?
But this may not be worth the effort. As all non-zero level gets
logarithmically smaller as we move bottom to top, so the time to insert nodes
in that level will not be high.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
