Re: unable to get more throughput with more threads

2017-03-24 Thread Suresh Pendap
Hi Shawn, It looks like you probably have pointed to the root cause of the issue. I am using a java client and using HttpClient library directly to fire the Http get queries. I am not using SolrJ client for firing the queries. The following is my code CloseableHttpClient httpclient =

Re: unable to get more throughput with more threads

2017-03-24 Thread Suresh Pendap
Erick, - I think I checked that my QueryResultsCache and DocumentCache ratios were close to 1. I will double check that by repeating my test. - I think checking the Qtimes in the log is a very good suggestion, I will also check that the next time I run my test - It is not possible as the client is

Re: unable to get more throughput with more threads

2017-03-24 Thread Shawn Heisey
On 3/23/2017 6:10 PM, Suresh Pendap wrote: > I performed the test with 1 thread, 10 client threads and 50 client > threads. I noticed that as I increased the number of threads, the > query latency kept increasing drastically which I was not expecting. What language and Solr library was the

Re: unable to get more throughput with more threads

2017-03-23 Thread Suresh Pendap
Edwin, The heap was not being used much, only 1GB of heap was being used out of 8GB. I do have space to allocate more to the heap size. I was reading in some SOLR performance blogs that it is better not to use large heap size, instead it is better to provide lot of space to the Operating system

Re: unable to get more throughput with more threads

2017-03-23 Thread Suresh Pendap
I am using version 6.3 of Solr On 3/23/17 7:56 PM, "Aman Deep Singh" wrote: >system

Re: unable to get more throughput with more threads

2017-03-23 Thread Erick Erickson
I'd check my I/O. Since you're firing the same query, I expect that you aren't I/O bound at all, since, as you say, the docs should already be in memory. This assumes that your document cache size is > 0. You can check this. Go to the admin UI, select one of your cores (not collection) and go to

Re: unable to get more throughput with more threads

2017-03-23 Thread Aman Deep Singh
You can play with the merge factor in the index config. If their is no frequent updates then make it 2 ,it will give you High throughput and less latency. On 24-Mar-2017 8:22 AM, "Zheng Lin Edwin Yeo" wrote: > I also did find that beyond 10 threads for 8GB heap size ,

Re: unable to get more throughput with more threads

2017-03-23 Thread Zheng Lin Edwin Yeo
I also did find that beyond 10 threads for 8GB heap size , there isn't much improvement with the performance. But you can increase your heap size a little if your system allows it. By the way, which Solr version are you using? Regards, Edwin On 24 March 2017 at 09:21, Matt Magnusson

Re: unable to get more throughput with more threads

2017-03-23 Thread Matt Magnusson
Out of curosity, what is your index size? I'm trying to do something similar with maximizing output, I'm currently looking at streaming expressions which I'm seeing some interesting results for, I'm also finding that the direct mass query route seems to hit a wall for performance. I'm also finding

unable to get more throughput with more threads

2017-03-23 Thread Suresh Pendap
Hi, I am new to SOLR search engine technology and I am trying to get some performance numbers to get maximum throughput from the SOLR cluster of a given size. I am currently doing only query load testing in which I randomly fire a bunch of queries to the SOLR cluster to generate the query load.