Hi Jude, I love the data-driven approach here!
Let’s be explicit about the change that brought this about: it’s multiThreaded=true https://issues.apache.org/jira/browse/SOLR-13350 Can you try setting https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html#indexSearcherExecutorThreads to -1 ? multiThreaded is supposed to perform well if there’s ample CPU. For users/benchmarks with lots of concurrent load, I don’t expect this option to perform well. It’s really a shame to see that the performance is impacted without even setting multiThreaded=true ! I think should disable this so it’s fully opt-in. ~ David > On Feb 10, 2025, at 2:19 PM, Jude Muriithi (BLOOMBERG/ 919 3RD A) > <jmuriit...@bloomberg.net> wrote: > > Greetings all, > > In the process of following up on this thread in the users mailing list: > https://lists.apache.org/thread/96ndn3n999dxpm71xjcwr4c6bjpd4sdy, we ran the > solr-benchmark project on various local development builds of Solr. For > context, this benchmark (https://github.com/pavel-chumakou/solr-benchmark) > repeats a set of 20 queries over an index of selected Wikipedia articles. > Running Solr locally, we saw the following results (all from a cold start): > > - Solr 9.8.0: Search time, ms: 742 473 390 375 381 307 380 327 297 334 304 > 386 342 315 286 308 311 285 306 430 > - Solr 9.7.0: Search time, ms: 767 479 398 331 314 308 289 296 336 339 324 > 325 286 288 294 288 307 299 399 295 > - Solr 9.6.1: Search time, ms: 328 59 7 8 6 4 8 6 2 2 1 2 1 2 2 2 1 1 2 2 > > After observing this discrepancy, we used the JFR to profile our local builds > of Solr while running the benchmark. This led us to the following discovery: > > - Build of upstream commit b8410234: Search time, ms: 1026 741 704 674 624 > 604 841 719 630 614 642 610 629 599 626 592 617 616 626 603 > - Build of the previous upstream commit (08a2bdd1): Search time, ms: 385 81 > 39 13 3 7 6 8 6 5 4 27 75 2 3 4 2 1 1 2 > > Without drawing any conclusions, our observations indicate that some change > related to the above commit is causing major performance degradation. This > would align with the various reports of performance degradation following the > release of Solr 9.7. > > All benchmarks above were run on an M3 Macbook Pro with Temurin Java 21.0.5. > To execute each benchmark, we ran the following commands: > > git clone https://github.com/pavel-chumakou/solr-benchmark.git > cd solr-benchmark > unzip wikitest.zip > > cd apache/solr > git checkout releases/solr/9.x.x > ./gradlew clean dev > cd solr/packaging/build/dev > cp -r ~/solr-benchmark/wikitest/ ./server/solr/wikitest/ > bin/solr start > > cd ~/solr-benchmark > java RunSolrQuery.java
