[
https://issues.apache.org/jira/browse/LUCENE-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475818#comment-17475818
]
Julie Tibshirani commented on LUCENE-10375:
-------------------------------------------
I was happily surprised it helped so much – each mapping operation didn't seem
that expensive. I'll be curious to dig into the nightly benchmark profiles
again once it's merged.
> Speed up HNSW merge by writing combined vector data
> ---------------------------------------------------
>
> Key: LUCENE-10375
> URL: https://issues.apache.org/jira/browse/LUCENE-10375
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Julie Tibshirani
> Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> When merging segments together, the HNSW writer creates a VectorValues
> instance that gives a merged view of all the segments' VectorValues. This
> merged instance is used when constructing the new HNSW graph. Graph building
> needs random access, and the merged VectorValues support this by mapping from
> merged ordinals -> segments and segment ordinals.
> This mapping seems to add overhead. The nightly indexing benchmarks sometimes
> show substantial time in Arrays.binarySearch (used to map an ordinal to a
> segment):
> https://blunders.io/jfr-demo/indexing-1kb-vectors-2022.01.09.18.03.19/top_down_cpu_samples.
> Instead of using a merged VectorValues to create the graph, maybe we could
> first write all the segment vectors to a file, and use that file to build the
> graph.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
