LantaoJin commented on PR #16794: URL: https://github.com/apache/lucene/pull/16794#issuecomment-6093627288
> @LantaoJin Overall I like the approach, it is safe, very closely follows the serial/incremental design (which your latest benchmarks also confirmed). But also there is too many barriers/synchronization 2 * Graphs size. That's why we don't see significant speedups, but I guess 1.31x and 1.17x is also good. Barriers cost little in my measurements. Synthetic, 8 workers: the new path is about 93% parallel-efficient (41.7 s at 1 worker / 8 = 5.2 s ideal, against 5.6 s), the old path about 99%. That is roughly 6%, not the main limit. I will enable `InfoStream` to measure barrier idle time. > Have you experimented with less barriers (e.g. to give each worker its own graph to completely work on), and comparing Recall/QPS graphs with the approach in this PR? "Each worker joins its own graph" may have two risks: 1. I already saw that order matters a lot. Inserting all join sets first dropped recall from 0.9852 to 0.9658 (benchmarked). 2. With few or skewed graphs, one worker owns most of the work. It could be a load balance issue. I will prototype the per-graph variant, plus a one-barrier-per-graph variant (overlap graph g's other nodes with graph g+1's join set) and compare force merge time and recall/visited sweeps on the same luceneutil inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
