Hi Solr Users, What could cause some nodes in a cluster to have high query latency when compared to the rest?
It is Solr 9.1.1 cluster of 32 nodes. Cluster has one collection divided into 32 shards and replication 3, so each node has one Solr server hosting 3 replicas. It is composite router collection so documents are evenly distributed across collection. Node roles are default so all the nodes are coordinator nodes and data nodes. The leader nodes are spread across the nodes. Should every shard not work the same and have close to same latency? Looks like 3 of 32 nodes consistently display high latency ( 1-2 seconds QTime) compared to the rest of the nodes(~30 ms). What can cause this behavior?