mayya-sharipova commented on PR #16664:
URL: https://github.com/apache/lucene/pull/16664#issuecomment-5636185836
@costin Thanks for tackling concurrent merges.
Before you invest more here: I'm not convinced by this approach. The
sequential algorithm was designed around the join set, which gives two
guarantees — coverage and ordering. `CompletedNeighborEps` correctly replaces
ordering, but nothing replaces coverage: a node takes the reduced beam off
whichever single neighbor happens to be finished, so anchor quality is
arbitrary and varies run to run. We keep the half of the algorithm that spends
less and drop the half that paid for it.
I'm also concerned about the extra heap: ~12 bytes per vector retained for
the whole graph build
---
I like the goal, but I think the shape is wrong, and both problems trace
back to partitioning work by *merged* ordinal ranges — a worker then no longer
knows which graph its node came from, hence the reverse-lookup tables and the
lost
guarantee.
We can explore an alternative approach to partition per source graph
instead, with two concurrent phases and a barrier: join set at full beam, then
the remainder with eps. That keeps both guarantees, makes the decision
deterministic, and removes the per-vector tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
