Could it be related to https://solr.apache.org/news.html#java-17-bug-affecting-solr ? Doubt it as you don't use much caching, but hotspot optimization of caches are disabled by default in 9.1. You could try to edit bin/solr script to disable the patch and see if anything is faster - risking a segfault crash instead :)
Jan > 2. des. 2022 kl. 10:11 skrev Joe Jones (DHCW - Software Development) > <joe.jo...@wales.nhs.uk.INVALID>: > > Hello all, > > We currently have a Solr cloud set up under version 5.4.1 running on Windows. > > It contains 45 million records in a collection split across 12 shards with a > replication factor of 2. The 12 shards are hosted across 6 servers running 4 > nodes each. 3 servers are in one data center and 3 in another so that we can > have high availability and site redundancy. > > This works great, serves hundreds of queries per minute and handles a trickle > feed of new documents (20K a day) and updates easily, but runs on old kit. > > We've built a new cloud using Solr 9.1 (Eclipse Adoptium OpenJDK 17.0.4) and > the same topology and settings on new servers and I have found that initial > few queries on a node are really bad. P95 times across the board are 6-8K > ms. It's like the first couple of queries are spinning up a new indexer > before settling into memory. Once in memory (or spun up), queries return in > <10ms, but leave it idle for 90 seconds and we get the slow query again. > I've noticed on some occasions the admin UI panel can be slow on query if > left idle. Using dev tools in the browser will also show for the first query > the page time is also greater than the already high query/elapsed time. > > We do not autowarm caches due to the nature of the queries being used. > Garbage collection looks fine on 1200mb allocation. They are quick and > infrequent, usually getting to 70% of 1200mb allocated before being cleaned. > > We see the same behaviour without any active indexing taking place so can > rule that out. The cloud was previously built on version 9 with the same > issue. > > > The only thing that helps keep queries quick is if I spam requests at the > cloud constantly. I can't compare any idle periods on the 5.4 Solr cloud > because there's always some activity. > > Perhaps once this is used in production the activity will keep things alive, > but is there something else I can look at to keep the cloud active at all > times? > > Thanks. > > Rydym yn croesawu derbyn gohebiaeth yng Nghymraeg. Byddwn yn ateb y fath > ohebiaeth yng Nghymraeg ac ni fydd hyn yn arwain at oedi. > We welcome receiving correspondence in Welsh. We will reply to such > correspondence in Welsh and this will not lead to a delay.
