Ok we think we found out the issue here. When solrcloud is started without specifying numShards argument solrcloud starts with a single shard but still thinks that there are multiple shards, so it forwards every single query to all the nodes in the cloud. We did a tcpdump on the node where queries are not targeted and found out that it is receiving POST requests from the node where queries are started. *&start=0&fsv=true&distrib=false&isShard=true&shard.url=serve1.com *
We solved the issue by explicitly adding numShards=1 argument to the solr start up script. Is this a bug? Re: SolrCloud - Query performance degrades with multiple servers Dec 06, 2012; 3:13pm — by sausarkar I also did a test running a load directed to one single server in the cloud and checked the CPU usage of other servers. It seems that even if there are no load directed to those servers there is a CPU spike each minute. Did you also di this test on the SolrCloud, any observations or suggestions? In Reply To Re: SolrCloud - Query performance degrades with multiple servers Dec 05, 2012; 7:59pm — by Mark Miller-3 This is just the std scatter gather distrib search stuff solr has been using since around 1.4. There is some overhead to that, but generally not much. I've measured it at around 30-50ms for a 100 machines, each with 10 million docs a few years ago. So…that doesn't help you much…but FYI… - Mark On Dec 5, 2012, at 5:35 PM, sausarkar <[hidden email]> wrote: > We are using SolrCloud and trying to configure it for testing purposes, we > are seeing that the average query time is increasing if we have more than > one node in the SolrCloud cluster. We have a single shard 12 gigs > index.Example:1 node, average query time *~28 msec* , load 140 > queries/second3 nodes, average query time *~110 msec*, load 420 > queries/second distributed equally on three servers so essentially 140 qps > on each node.Is there any inter node communication going on for queries, > is > there any setting on the Solrcloud for query tuning for a cloud config > with > multiple nodes.Please help. > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/SolrCloud-Query-performance-degrades-with-multiple-servers-tp4024660.html > Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-Query-performance-degrades-with-multiple-servers-tp4024660p4024986.html Sent from the Solr - User mailing list archive at Nabble.com.
