vigyasharma commented on PR #15676: URL: https://github.com/apache/lucene/pull/15676#issuecomment-4007906452
Thank you @krickert for working through this. Your attention to testing and the work you've done on profiling different distributed scenario setups is admirable. Thank you, for making it all open source! <3 This is an interesting idea and I'm curious to see how the profiling fares out. Going over the issue thread, it seems that one persistent problem is that we don't know when we are in an optimal graph neighborhood to start applying the externally imposed threshold. Applying it too early gives bad results (obv because we haven't reached the good neighborhood yet). And relying on static no. of hops or iterations seems wasteful. I'm curious about ideas for this specific sub-problem. Would it help to look at the variance in similarity scores for all nodes in the candidates neighbor array during graph search? Perhaps variance would be high initially (in bad neighborhood) but slowly fall as we reach better graph areas? It's not super reliable though because we could be in a region where all neighbors are similarly bad.. so maybe we want to also factor in the similarity scores we're seeing? Another heuristic could be to apply the external threshold once we have at least a few candidates that survive graph traversal iterations. These nodes are potentially good results and applying the external threshold after collecting _some_ good results means we are only short-circuiting the long tail of collected results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
