debe commented on PR #1155: URL: https://github.com/apache/solr/pull/1155#issuecomment-1344368444
Hi @stillalex, I've changed your proposed benchmark a little bit. See [main](https://github.com/apache/solr/compare/main...tboeghk:solr:main-bench?expand=1) and [PR1155](https://github.com/tboeghk/solr/compare/main...tboeghk:solr:issues/solrcores-finer-locking-bench?expand=1). Main points are: 1. I think you actually need to do something while holding the lock like it is in reality. So I've tried to mimic the behaviour of open() and close(). I set the parameter incRefCount to true in order to do something and I set doAnswer on SolrCore mock but it didn't work as expected. Basically you can't call a method intercepted by mockito inside your benchmarking function, because it will ruin it with lots of GC. The reason for this behaviour are those method interceptors and stacktrace objects, which all want to be GCed. I avoided that by introducing a consumer to do some work while the lock is being held: ```java SolrCore getCoreFromAnyList(String name, boolean incRefCount, UUID coreId, Consumer<SolrCore> callback) { READ_WRITE_LOCK.readLock().lock(); try { SolrCore core = getCoreFromAnyList(name, incRefCount, coreId); callback.accept(core); return core; } finally { READ_WRITE_LOCK.readLock().unlock(); } } ``` 2. I think one should limit the number of threads to the number of CPU cores, otherwise you're benchmarking context switching of OS threads. To get an overview which public methods from SolrCores and CoreContainer are actually called and how frequent, I've checked a recent flight recorder image from one of our production machines. I was wondering if benchmarking getCoreFromAnyList() is sufficient to model the behaviour of our system. This is a 20 Minute frame without creation of a new Core. Here the distribution is in relation to the number of request calls. 1. Requests 100% 2. CoreContainer.getCore(String) 57% <--- this calls solrCores.getCoreFromAnyList 4. CoreContainer.getRequestHandler(String) 36% <--- doesn't lock 5. SolrCores.getCoreDescriptor(String) 12% <--- locks 6. CoreContainer.getCore(String, UUID) 9% <--- this calls solrCores.getCoreFromAnyList Here are my benchmark result runs: The synchronized one: ``` $ cat results-synchronized.txt running JMH with args: -prof jfr:dir=profile-results;configName=jfr-profile.jfc SolrCoresBenchTest # Detecting actual CPU count: 12 detected # JMH version: 1.32 # VM version: JDK 17.0.2, OpenJDK 64-Bit Server VM, 17.0.2+8-jvmci-22.0-b05 # VM invoker: /home/db/.sdkman/candidates/java/22.0.0.2.r17-grl/bin/java # VM options: -Djmh.shutdownTimeout=5 -Djmh.shutdownTimeout.step=3 -Djava.security.egd=file:/dev/./urandom -XX:-UseBiasedLocking -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints --add-opens=java.base/java.lang.reflect=ALL-UNNAMED -Dlog4jConfigurationFile=./log4j2-bench.xml -Dlog4j2.is.webapp=false -Dlog4j2.garbagefreeThreadContextMap=true -Dlog4j2.enableDirectEncoders=true -Dlog4j2.enable.threadlocals=true -XX:+UseG1GC -XX:+ParallelRefProcEnabled # Blackhole mode: full + dont-inline hint # Warmup: 3 iterations, 10 s each # Measurement: 4 iterations, 20 s each # Timeout: 300 s per iteration # Threads: 12 threads, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList # Run progress: 0,00% complete, ETA 00:01:50 # Fork: 1 of 1 # Preparing profilers: JavaFlightRecorderProfiler OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release. # Warmup Iteration 1: --> benchmark random seed: 6624420638116043983 1835172,373 ops/s # Warmup Iteration 2: 1821486,883 ops/s # Warmup Iteration 3: 1835960,182 ops/s Iteration 1: 1650271,601 ops/s Iteration 2: 1662880,927 ops/s Iteration 3: 1661900,359 ops/s Iteration 4: 1661879,729 ops/s ·jfr: (text only) # Processing profiler results: JavaFlightRecorderProfiler Result "org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList": 1659233,154 ±(99.9%) 38724,163 ops/s [Average] (min, avg, max) = (1650271,601, 1659233,154, 1662880,927), stdev = 5992,607 CI (99.9%): [1620508,991, 1697957,317] (assumes normal distribution) Secondary result "org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList:·jfr": JFR profiler results: profile-results/org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList-Throughput/profile.jfr ``` The rwlock one: ``` $ cat results-rwlock.txt running JMH with args: -prof jfr:dir=profile-results;configName=jfr-profile.jfc SolrCoresBenchTest # Detecting actual CPU count: 12 detected # JMH version: 1.32 # VM version: JDK 17.0.2, OpenJDK 64-Bit Server VM, 17.0.2+8-jvmci-22.0-b05 # VM invoker: /home/db/.sdkman/candidates/java/22.0.0.2.r17-grl/bin/java # VM options: -Djmh.shutdownTimeout=5 -Djmh.shutdownTimeout.step=3 -Djava.security.egd=file:/dev/./urandom -XX:-UseBiasedLocking -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints --add-opens=java.base/java.lang.reflect=ALL-UNNAMED -Dlog4jConfigurationFile=./log4j2-bench.xml -Dlog4j2.is.webapp=false -Dlog4j2.garbagefreeThreadContextMap=true -Dlog4j2.enableDirectEncoders=true -Dlog4j2.enable.threadlocals=true -XX:+UseG1GC -XX:+ParallelRefProcEnabled # Blackhole mode: full + dont-inline hint # Warmup: 3 iterations, 10 s each # Measurement: 4 iterations, 20 s each # Timeout: 300 s per iteration # Threads: 12 threads, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList # Run progress: 0,00% complete, ETA 00:01:50 # Fork: 1 of 1 # Preparing profilers: JavaFlightRecorderProfiler OpenJDK 64-Bit Server VM warning: Option UseBiasedLocking was deprecated in version 15.0 and will likely be removed in a future release. # Warmup Iteration 1: --> benchmark random seed: 6624420638116043983 2415802,757 ops/s # Warmup Iteration 2: 2426055,238 ops/s # Warmup Iteration 3: 2417938,908 ops/s Iteration 1: 2342669,342 ops/s Iteration 2: 2360751,931 ops/s Iteration 3: 2343344,162 ops/s Iteration 4: 2363582,478 ops/s ·jfr: (text only) # Processing profiler results: JavaFlightRecorderProfiler Result "org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList": 2352586,978 ±(99.9%) 71895,415 ops/s [Average] (min, avg, max) = (2342669,342, 2352586,978, 2363582,478), stdev = 11125,895 CI (99.9%): [2280691,563, 2424482,393] (assumes normal distribution) Secondary result "org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList:·jfr": JFR profiler results: profile-results/org.apache.solr.core.SolrCoresBenchTest.getCoreFromAnyList-Throughput/profile.jfr ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
