Re: Do Multiprocessing on Solr to search?
On 2/25/2015 9:31 AM, Nitin Solanki wrote: I want to search lakhs of queries/terms concurrently. Is there any technique to do multiprocessing on Solr? Is Solr is capable to handle this situation? I wrote a code in python that do multiprocessing and search lakhs of queries and do hit on Solr simultaneously/ parallely at once but it seems that Solr doesn't able to handle queries at once. Any help Please? Solr is fully multi-threaded and capable of handling multiple requests simultaneously. Any of the common servlet containers that are typically used to run Solr are *also* fully multi-threaded, but may require configuration adjustment to allow more threads. The jetty install that comes with the Solr example server is tuned to allow 1 threads. Even if you have a very well-tuned Solr install on exceptionally robust hardware, I would not expect a single index on a single server to be able to handle more than a few hundred requests per second. If you need hundreds of thousands of simultaneous queries, you're going to need a lot of replicas on a lot of servers. With that volume you would want a load balancer to direct requests to those replicas. You may also run into problems related to TCP port exhaustion. Thanks, Shawn
RE: Do Multiprocessing on Solr to search?
Nitin Solanki [nitinml...@gmail.com] wrote: I want to search lakhs of queries/terms concurrently. Is there any technique to do multiprocessing on Solr? Each concurrent search in Solr runs in its own thread, so the answer is yes, it does so out of the box with concurrent searches. Is Solr is capable to handle this situation? Yes and no. There is a limit to the number of concurrent connections and as far as I remember, it is 10.000 out of the box. If you are using SolrCloud, deadlocks might happen if you exceed the limit. Anyway, I would not recommend running 10.000 concurrent searches as it leads to congestion. You will probably get a higher throughput by queueing your requests and process then with 100 concurrent searches or so. Do test. - Toke Eskildsen
Do Multiprocessing on Solr to search?
Hello, I want to search lakhs of queries/terms concurrently. Is there any technique to do multiprocessing on Solr? Is Solr is capable to handle this situation? I wrote a code in python that do multiprocessing and search lakhs of queries and do hit on Solr simultaneously/ parallely at once but it seems that Solr doesn't able to handle queries at once. Any help Please?
