msokolov commented on PR #16805: URL: https://github.com/apache/lucene/pull/16805#issuecomment-6155775704
Let me summarize the changes here. The main idea is to cut over all the single-threaded Collectors used in the join module to use multi-threaded collection (CollectorManager + multiple Collectors). The join module is gathering join keys and scores across matching documents in order to generate a query that joins with those keys (and uses the scores for scoring?). We could do this in one of two ways: (1) gather into segment-specific data structures and then merge those in CollectorManager.reduce, or (2) gather into shared concurrency-safe data structures. I basically tried to do this with the smallest change to code, opting to use concurrent data structures if it made the migration job simpler. It's possible we may lose some performance in the left-hand side of these joins since for example I replaced hppc collections with java.util collections that box their values, and because there can be some lock contention and bookkeeping overhead of collecting into shared concurrent data structures. I did some simple perform ance testing by timing the join collectors while running unit tests, and the overall speed improved, I guess due to concurrency, so at least things did not get horrible. In `JoinUtil` we define aggregation lambdas that used to collect into hppc maps, but I switched to collect into ConcurrentHashMap, for thread safety. So this is method (1) above. In `GenericTermsCollector` we collect into per-segment score arrays and BytesRefHash collections, and then merge them , re-using the BytesRefHash from the first collector, and carefully remapping scores into the proper array locations corresponding with the terms ordinals (since terms duplicated across segments need to be merged). This was approach (1) above. Finally `GlobalOrdinalsWithScoreCollectorManager` collects scores and counts into multiple primitive arrays. I left this alone since the arrays are indexed by global docid and simply added locks around each block to synchronize access while writing. Reads by consumers can be done without any locking since no writing happens after collection is complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
