xiangfu0 commented on code in PR #20072:
URL: https://github.com/apache/pinot/pull/20072#discussion_r4419531481
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/vector/HnswVectorIndexCreator.java:
##########
@@ -63,6 +96,18 @@ public HnswVectorIndexCreator(String column, File
segmentIndexDir, VectorIndexCo
_vectorSimilarityFunction =
VectorIndexUtils.toSimilarityFunction(vectorIndexConfig.getVectorDistanceFunction());
_storeInSegmentFile = vectorIndexConfig.isStoreInSegmentFile();
_segmentIndexDir = segmentIndexDir;
+ Map<String, String> properties = vectorIndexConfig.getProperties();
+ // Clamped as well as validated, since configs written before validation
covered these keys skip it
+ _buildThreads = Math.min(VectorIndexConfigValidator.MAX_HNSW_BUILD_THREADS,
+ Math.max(1, Integer.parseInt(properties.getOrDefault("buildThreads",
"1"))));
+ _forceMerge = Boolean.parseBoolean(properties.getOrDefault("forceMerge",
"true"));
+ String parallelBuildMaxDocs = properties.get("parallelBuildMaxDocs");
+ _parallelBuildMaxDocs = parallelBuildMaxDocs != null ?
Integer.parseInt(parallelBuildMaxDocs)
+ : (int) Math.max(1, Math.min(DEFAULT_PARALLEL_BUILD_MAX_DOCS,
+ DEFAULT_PARALLEL_BUILD_MAX_BYTES / ((long) Float.BYTES *
_vectorDimension)));
+ _bufferedVectors = _buildThreads > 1 ? new ArrayList<>() : null;
+ _mergeExecutor = _buildThreads > 1 && _forceMerge ?
Executors.newFixedThreadPool(_buildThreads) : null;
Review Comment:
**MINOR:** With concurrent merge workers, the merged graph depends on thread
timing. Three builds of the same 50k vectors gave three different result
fingerprints, with recall@10 of 0.898–0.902 vs 0.923 for base.
`forceMerge=false` and `buildThreads=1` were identical every run.
`SegmentPreProcessor` builds this index locally on each server, so replicas can
then return different top-K for the same query. Please document this tradeoff,
or keep `forceMerge=true` deterministic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
