[
https://issues.apache.org/jira/browse/LUCENE-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410532#comment-17410532
]
ASF subversion and git services commented on LUCENE-10054:
----------------------------------------------------------
Commit a2c8f2805e18895bdfcca568126353c86b95e50b in lucene's branch
refs/heads/hnsw from Mayya Sharipova
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=a2c8f28 ]
LUCENE-10054 Handle hierarchy in graph construction and search (#267)
This patch handles hierarchy in graph construction and search,
but only in memory.
Changes:
- HnswGraphBuilder has an extra parameter :ml normalization factor for level
generation. When ml=0, the graph will have only a single layer: SNW.
When ml > 0, the graph will have multiple layers.
- The recommended ml value from 2018 HNSW paper is :
ml = 1 / Math.log(1.0 * maxConn), which was used for tests in
TestHnswGraph class.
- When ml = 0, we use the previous code in the method buildSNW
- When ml > 0, the method buildHNSW is used, that according the paper,
for every new node: generates a random level for this node, and then
places the new node in those levels with its connections.
- HnswGraph's search method has also been modified to handle two cases:
- When ml = 0, we use the previous code for a flat graph: generate
boundedNumSeed number of random entry points, and use them to search
a flat graph
- When ml > 0, we use the hierarchical graph: from max level till level 1
using topK=1, and on the level 0th using topK=boundedNumSeed
Work left for future: handle hierarchy on disk
> Handle hierarchy in HNSW graph
> ------------------------------
>
> Key: LUCENE-10054
> URL: https://issues.apache.org/jira/browse/LUCENE-10054
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Mayya Sharipova
> Priority: Major
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> Currently HNSW graph is represented as a single layer graph.
> We would like to extend it to handle hierarchy as per
> [discussion|https://issues.apache.org/jira/browse/LUCENE-9004?focusedCommentId=17393216&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17393216].
>
>
> TODO tasks:
> - add multiple layers in the HnswGraph class
> - modify the format in Lucene90HnswVectorsWriter and
> Lucene90HnswVectorsReader to handle multiple layers
> - modify graph construction and search algorithm to handle hierarchy
> - run benchmarks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
