dnhatn commented on code in PR #12216:
URL: https://github.com/apache/lucene/pull/12216#discussion_r1152745572
##########
lucene/core/src/java/org/apache/lucene/index/ConcurrentApproximatePriorityQueue.java:
##########
@@ -22,25 +22,40 @@
/**
* Concurrent version of {@link ApproximatePriorityQueue}, which trades a bit
more of ordering for
- * better concurrency by maintaining 8 sub {@link ApproximatePriorityQueue}s
that are locked
- * independently.
+ * better concurrency by maintaining multiple sub {@link
ApproximatePriorityQueue}s that are locked
+ * independently. The number of subs is computed dynamically based on hardware
concurrency.
*/
final class ConcurrentApproximatePriorityQueue<T> {
- /** Keeping 8 queues should already help a lot compared to a single one. */
- static final int CONCURRENCY = 8;
+ static Integer CONCURRENCY_OVERRIDE;
- private static final int MASK = 0x07;
+ private static final int getConcurrency() {
+ int concurrency;
+ if (CONCURRENCY_OVERRIDE != null) {
+ concurrency = CONCURRENCY_OVERRIDE;
+ } else {
+ int coreCount = Runtime.getRuntime().availableProcessors();
+ // Aim for ~4 entries per slot when indexing with one thread per CPU
core. The trade-off is
+ // that if we set the concurrency too high then we'll completely lose
the bias towards larger
+ // DWPTs. And if we set it too low then we risk seeing contention.
+ concurrency = coreCount / 4;
+ }
+ concurrency = Math.max(1, concurrency);
+ concurrency = Math.min(256, concurrency);
+ return concurrency;
+ }
+ final int concurrency;
final Lock[] locks;
final ApproximatePriorityQueue<T>[] queues;
ConcurrentApproximatePriorityQueue() {
- locks = new Lock[CONCURRENCY];
+ concurrency = getConcurrency();
Review Comment:
Can we have another constructor accepting the `concurrency` level, and the
default constructor using the default computed value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
