I am running Solr 4.5.1. Here is how my setup looks:
Have 2 modest sized Collections.
Collection 1 - 2 shards, 3 replicas (Size of Shard 1 - 115
MB, Size of Shard 2 - 55 MB)
Collection 2 - 2 shards, 3 replicas (Size of Shard 2 - 3.5
GB, Size of Shard 2 - 1 GB)
These two collections are distributed across:
6 Tomcat Nodes setup on 3 VMs (2 Nodes per VM)
Each of the 6 Tomcat nodes has a XmS / XmX setting of 2 GB
Each of the 3 VMs have a Physical Memory (RAM) of 32 GB
As you can see my Collections are pretty small - This is actually a test
environment (and NOT Production), However my users (only have a handful of
testers) are complaining of sporadic performances issues on the Search.
Here are my observations from the application logs:
1) Out of 200 sample searches across both collections - 13 requests are slow
(3 slow responses on Collection 1 and 10 slow responses on Collection 2).
2) When things run fast - they are really fast (Qtimes of 25 - 100
milliseconds) - but when things are slow - I can see that the QTime
consistently hovers around the 5 second (or 5000 millisecond mark). I am
seeing responses of the order of 5024, 5094, 5035 ms - as though something
just hung for 5 seconds. I am observing this 5 second delay on both
Collections - which I feel is unusual - because both contain very different
data sets. I am unable to figure out whats causing the QTime to be so
consistent around the 5 second mark.
3) I build my index only once. I did try running an optimize on both
Collection 1 and Collection 2 after the users complained - I did notice that
post the optimize the segment count on each of the four shards did come down
- but that still didn't resolve the slowness on the searches (I was hoping
it would).
4) I am looking at the Solr Dashboard for more clues - My TomCat nodes are
definitely NOT running out of memory - the 6 nodes are consuming anywhere
between 500 MB - 1 GB RAM
5) The File Descriptor counts are under control - can only see a maximum of
100 file descriptors being used of a total of 4096
6) The Solr dashboard is however showing that 0.2% (or 9.8MB) of Swap Space
being consumed on one of the 3 VMs. Is this a concern ?
7) Also looked at the Plugin / Stats for every core on the Solr Dashboard. I
can't see any evictions happening in any of the caches - Its always ZERO.
Has anyone encountered such an issue ? What else should I be looking for to
debug my problem ?
Thanks
--
View this message in context:
http://lucene.472066.n3.nabble.com/Slow-QTimes-5-seconds-for-Small-sized-Collections-tp4143681.html
Sent from the Solr - User mailing list archive at Nabble.com.
