Re: Some performance questions....

> On Mar 24, 2018, at 5:21 PM, Deepak Goel wrote: > > My first test was to test with static queries. Does Solr scale-up as we > increase the load of same query? > > The second test would be to check with 'Different Queries'. > > And then finally check with 80% similar

Re: Some performance questions....

Some observations: *#* The CPU Load on idxa1 never crosses above 91% mark mostly even if you increase the load (by increasing the number of threads). This is similar to my environment (I can never cross 90% on Linux even if I increase the load. For Windows I can never cross 65% for some reason)

Re: Some performance questions....

On 3/25/2018 7:15 AM, Deepak Goel wrote: $ Why is the 'qps' not increasing with increase in threads? (If I understand the qps parameter right?) Likely because I sent all these queries to a single copy of the index.  We only have two copies of the index in production, plus a third copy on a

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sun, Mar 25, 2018 at 2:24 PM, Shawn Heisey wrote: > On

Re: Some performance questions....

On 3/25/2018 1:45 AM, Shawn Heisey wrote: I have written a little test program that can pound the system harder, need a little more time to gather what I learned with it. Here's the code and three results with different threadcounts:

Re: Some performance questions....

On 3/24/2018 10:42 PM, Deepak Goel wrote: I believe you ran this query with a 1 user load. Or was it a multi-user load test? If it was multi-user load test, how many users did you test for? And what were the utilisations and tps? It was late Saturday night when I did that.  There's almost no

Re: Some performance questions....

On 25 Mar 2018 6:49 am, "Shawn Heisey" wrote: On 3/24/2018 6:21 PM, Deepak Goel wrote: > Do you have any documented proof of the same (1 to 5ms)? Or is it an > educated guess > Just now, I did a test. I did a "*:*" query (all docs), the QTime was 194 milliseconds,

Re: Some performance questions....

On 3/24/2018 6:21 PM, Deepak Goel wrote: Do you have any documented proof of the same (1 to 5ms)? Or is it an educated guess Just now, I did a test.  I did a "*:*" query (all docs), the QTime was 194 milliseconds, numFound was 188635489.  Then I did the exact same query again.  QTime dropped

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sun, Mar 25, 2018 at 4:00 AM, Shawn Heisey wrote: > On

Re: Some performance questions....

On 3/24/2018 1:25 PM, Deepak Goel wrote: Please check the section *Questions from ‘Around the World’* in the following doc for answers to your questions: *https://docs.google.com/document/d/1ZwyveG-Zjy7tbsvh9xjMug4bnIMRqKnNax3jh4GJlzM/edit?usp=sharing The document says that 80 percent of the

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sat, Mar 24, 2018 at 6:03 AM, Rick Leir wrote: > > > Deep, >

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sat, Mar 24, 2018 at 5:16 AM, Shawn Heisey wrote: > On

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sat, Mar 24, 2018 at 6:18 AM, Shawn Heisey wrote: > On

Re: Some performance questions....

On 3/23/2018 1:13 PM, Deepak Goel wrote: > Yes I am now creating a client object only once. On Linux it has superb > results (performance improves by around two times). However on Windows it > has no improvement > > *SoftwareThroughput (/sec)Response Time (msec)Utilization (%CPU)UnTuned >

Re: Some performance questions....

Deep, What is the test so I can try it. 75 or 90 ms .. is that the JVM startup time? Cheers -- Rick >> >> >I have stated the numbers which I found during my test. The best way to >verify them is for someone else to run the same test. Otherwise I don't >see >how we can verify the results --

Re: Some performance questions....

On 3/23/2018 11:31 AM, Deepak Goel wrote: > Do you have any specific questions about the benchmark setup? How many docs are in the Solr index?  How much disk space does it consume?  How much total memory is in the machine?  How much memory is allocated to Java heaps?  Is there any other software

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Fri, Mar 23, 2018 at 11:38 PM, Shawn Heisey wrote: > On

Re: Some performance questions....

On 3/23/2018 11:21 AM, Deepak Goel wrote: >> I tried the above suggestion. The throughput and utilisation remain the >> same (they dont increase even if I increase the load). The response time >> comes down. >> Are you still creating a new client object for every query?  Changing how the client

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Tue, Mar 20, 2018 at 3:32 AM, Shawn Heisey wrote: > On

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Thu, Mar 22, 2018 at 1:25 AM, Deepak Goel wrote: > > > > > Deepak

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Mon, Mar 19, 2018 at 2:40 AM, Walter Underwood wrote: > > On

Re: Some performance questions....

Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Sat, Mar 17, 2018 at 2:56 AM, Shawn Heisey wrote: > On

Re: Some performance questions....

On 3/16/2018 4:24 PM, Deepak Goel wrote: > It is taking less than 100ms to create a HttpSolrClient Object "Less than 100ms" is vague.  Let's say by that you mean it takes at least 50 milliseconds.  This is a lot slower than I expected it to be, but if you've measured it, I'll accept that. If

Re: Some performance questions....

> On Mar 17, 2018, at 3:23 AM, Deepak Goel wrote: > > Sorry for being rude. But the ' results ' please, not the ' road to the > results ' We have 15 different search collections, all different sizes and all with different kinds of queries. Here are the two major ones. 22

Re: Some performance questions....

Looks like I've opened up a very interesting can of worms Thank you to all that are posting to this thread, I'm learning a lot... The way I see it now... a Single Solr instance on this machine, seems like the most intelligent choice. And then as upgrade path, adding in-expensive machines.

Re: Some performance questions....

On 17 Mar 2018 05:19, "Walter Underwood" wrote: > On Mar 16, 2018, at 3:26 PM, Deepak Goel wrote: > > Can you please post results of your test? > > Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource I could, but it probably would not

Re: Some performance questions....

> On Mar 16, 2018, at 3:26 PM, Deepak Goel wrote: > > Can you please post results of your test? > > Please tell us the tps at 25%, 50%, 75%, 100% of your CPU resource I could, but it probably would not be useful for your documents or your queries. We have 22 million

Re: Some performance questions....

On Sat, Mar 17, 2018 at 3:11 AM, Walter Underwood wrote: > > On Mar 16, 2018, at 1:21 PM, Deepak Goel wrote: > > > > However a single client object with thousands of queries coming in would > > surely become a bottleneck. I can test this scenario too. >

Re: Some performance questions....

On Sat, Mar 17, 2018 at 2:56 AM, Shawn Heisey wrote: > On 3/16/2018 2:21 PM, Deepak Goel wrote: > > I wanted to test how many max connections can Solr handle concurrently. > > Also I would have to implement an 'connection pooling' of the > client-object > > connections

Re: Some performance questions....

> On Mar 16, 2018, at 1:21 PM, Deepak Goel wrote: > > However a single client object with thousands of queries coming in would > surely become a bottleneck. I can test this scenario too. No it isn’t. The single client object is thread-safe and manages a pool of connections.

Re: Some performance questions....

On 3/16/2018 2:21 PM, Deepak Goel wrote: > I wanted to test how many max connections can Solr handle concurrently. > Also I would have to implement an 'connection pooling' of the client-object > connections rather than a single connection thread > > However a single client object with thousands of

RE: Some performance questions....

ak Goel [mailto:deic...@gmail.com] Sent: Friday, March 16, 2018 4:22 PM To: solr-user@lucene.apache.org Subject: Re: Some performance questions On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 3/16/2018 7:38 AM, Deepak Goel wrote: > > I did a performanc

Re: Some performance questions....

On Sat, Mar 17, 2018 at 1:06 AM, Shawn Heisey wrote: > On 3/16/2018 7:38 AM, Deepak Goel wrote: > > I did a performance study of Solr a while back. And I found that it does > > not scale beyond a particular point on a single machine (could be due to > > the way its coded).

Re: Some performance questions....

On 3/16/2018 7:38 AM, Deepak Goel wrote: > I did a performance study of Solr a while back. And I found that it does > not scale beyond a particular point on a single machine (could be due to > the way its coded). Hence multiple instances might make sense. > >

Re: Some performance questions....

> That benchmark is on Windows, so not interesting for most of us. I guess I must have missed this in the author's question. Did he describe his OS? Also other applications scale well on Windows. Why would Solr be different? The Solr page does not say about any performance limits on windows

Re: Some performance questions....

> On Mar 16, 2018, at 6:26 AM, Deepak Goel wrote: > > I would try multiple Solr instances rather a single Solr instance (it > definitely will give a performance boost) > I would avoid multiple Solr instances on single machine. I can use all 36 cores on our servers with one Solr

Re: Some performance questions....

On Mar 16, 2018, at 6:38 AM, Deepak Goel wrote: > > I did a performance study of Solr a while back. And I found that it does > not scale beyond a particular point on a single machine (could be due to > the way its coded). Hence multiple instances might make sense. > >

Re: Some performance questions....

> On Mar 16, 2018, at 6:26 AM, Deepak Goel wrote: > > I would try multiple Solr instances rather a single Solr instance (it > definitely will give a performance boost) I would avoid multiple Solr instances on single machine. I can use all 36 cores on our servers with one

Re: Some performance questions....

On Fri, Mar 16, 2018 at 6:03 PM, Shawn Heisey wrote: > On 3/15/2018 6:34 AM, BlackIce wrote: > >> However the main app that will be >> running is more or less a single threated app which takes advantage when >> run under several instances, ie: parallelism, so I thought,

Re: Some performance questions....

>I think there is no benefit in having multiple Solr instances on a single >server, unless the heap memory required by the JVM is too big. Deepak*** I would try multiple Solr instances rather a single Solr instance (it definitely will give a performance boost) Deepak*** >And remember that

Re: Some performance questions....

On 3/15/2018 6:34 AM, BlackIce wrote: However the main app that will be running is more or less a single threated app which takes advantage when run under several instances, ie: parallelism, so I thought, since I'm at it I may give solr a few instances as well Solr is a fully threaded app,

Re: Some performance questions....

*Single Solr Instance VS Multiple Solr instances on Single Server * I think there is no benefit in having multiple Solr instances on a single server, unless the heap memory required by the JVM is too big. And remember that this has relatively to do with the index size ( inverted index is memory

Re: Some performance questions....

Please see inline... Deepak "Please stop cruelty to Animals, help by becoming a Vegan" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" On Thu, Mar 15, 2018 at 6:04 PM, BlackIce

Re: Some performance questions....

Shawn: well the idea was to utilize system resources more efficiently.. this is not due so much to Solr, as I sayd I don't know that much about Solr, except Shema.xml and Solarconfig.xml - However the main app that will be running is more or less a single threated app which takes advantage when

Re: Some performance questions....

On 3/14/2018 5:49 AM, BlackIce wrote: I was just thinking Do I really need separate VM's in order to run multiple Solr instances? Doesn't it suffice to have each instance in its own user account? You can run multiple instances all under the same account on one machine.  But for a single

Re: Some performance questions....

Have you measured the overhead of VM anytime? Or have you read it somewhere? On 14 Mar 2018 18:10, "BlackIce" wrote: > but it should be possible, without the overhead of VM's > > On Wed, Mar 14, 2018 at 1:30 PM, Deepak Goel wrote: > > > The OS

Re: Some performance questions....

but it should be possible, without the overhead of VM's On Wed, Mar 14, 2018 at 1:30 PM, Deepak Goel wrote: > The OS resources would be shared in that case > > On 14 Mar 2018 17:19, "BlackIce" wrote: > > > I was just thinking Do I really need

Re: Some performance questions....

The OS resources would be shared in that case On 14 Mar 2018 17:19, "BlackIce" wrote: > I was just thinking Do I really need separate VM's in order to run > multiple Solr instances? Doesn't it suffice to have each instance in its > own user account? > > Greetz > > On

Re: Some performance questions....

I was just thinking Do I really need separate VM's in order to run multiple Solr instances? Doesn't it suffice to have each instance in its own user account? Greetz On Mon, Mar 12, 2018 at 7:41 PM, BlackIce wrote: > I don't have any production logs and this all

Re: Some performance questions....

I don't have any production logs and this all sounds to complicated. So, I'll just trow the system together in a way it makes the most sense for now.. collect some logs and then do some testing further down the road. For now just get the sucker up and running. Thanks all On Mon, Mar 12,

Re: Some performance questions....

I am not sure if I understand your question *"How do I test this?"* You have to run test (benchmark test) of transactions (queries) which are most representative of your system (requirement). You can use a performance testing tool like JMeter (along with PerfMon configured for utilisation

Re: Some performance questions....

So Im thinking following scenarios : Single instance with drives in raid 0, raid 10 and raid 5. And then having 3 Vms and 4 Solr instances each with its own HD. How do I test this? Greetz On Mar 12, 2018 1:16 PM, "BlackIce" wrote: > OK, so we're gone nowhere, since

Re: Some performance questions....

Benchmark with production logs. Replay them at a constant request rate. Measure the response time and look at the median and 90th or 95th percentile. Do not use the average response time, because that will be thrown off by outliers. It is best to run a few thousand warming queries before

Re: Some performance questions....

OK, so we're gone nowhere, since I've already lost lots of time... A few days more or less won't make a difference I'd be willing to benchmark if some tells me how to. Greetz On Mar 12, 2018 7:17 AM, "Deepak Goel" wrote: > Now you are mixing your original question

Re: Some performance questions....

On 3/12/2018 3:22 AM, Deepak Goel wrote: A single OS and JVM does not scale linearly for higher loads. If you have seperate OS and Java, the load is distributed across multiple instances (with each instance only requiered to support a smaller load and hence would scale nicely) I had found this

Re: Some performance questions....

Now you are mixing your original question about performance with reliability On 12 Mar 2018 02:29, "BlackIce" wrote: > Second to this wouldn't 4 Solr instances each with its own HD be fault > tolerant? vs. one solr instance with 4 HD's in RAID 0? Plus to his comes > the

Re: Some performance questions....

We need benchmarks or data to support the claim. A single OS and JVM does not scale linearly for higher loads. If you have seperate OS and Java, the load is distributed across multiple instances (with each instance only requiered to support a smaller load and hence would scale nicely) I had

Re: Some performance questions....

On 3/11/2018 7:39 PM, Deepak Goel wrote: I doubt this. It would be great if someone can subtantiate this with hard facts This seems to be in response to my claim that virtualization always has overhead.  I don't see how this statement can be at all controversial. Virtualization isn't free,

Re: Some performance questions....

On 12 Mar 2018 05:51, "Shawn Heisey" wrote: On 3/11/2018 11:35 AM, BlackIce wrote: > I have some questions regarding performance. > > Lets says I have a dual CPU with a total of 8 cores and 24 GB RAM for my > Solr and some other stuff. > > Would it be more beneficial to

Re: Some performance questions....

On 3/11/2018 11:35 AM, BlackIce wrote: I have some questions regarding performance. Lets says I have a dual CPU with a total of 8 cores and 24 GB RAM for my Solr and some other stuff. Would it be more beneficial to only run 1 instance of Solr with the collection stored on 4 HD's in RAID 0??

Re: Some performance questions....

Second to this wouldn't 4 Solr instances each with its own HD be fault tolerant? vs. one solr instance with 4 HD's in RAID 0? Plus to his comes the storage capacity, I need the capacity of those 4 drives... the more I read.. the more questions On Sun, Mar 11, 2018 at 9:43 PM, BlackIce

Re: Some performance questions....

Thnx for the pointers. I haven't given much thought to Solr, asides shemal.xml and solrconfig.xml and I'm just diving into a bit more deeper stuff! Greetz RRK On Sun, Mar 11, 2018 at 8:58 PM, Deepak Goel wrote: > To rephrase your Question > > "Does Solr do well with

Re: Some performance questions....

To rephrase your Question "Does Solr do well with Scale-up or Scale-out?" Are there any Performance Benchmarks for the same out there supporting the claim? On 11 Mar 2018 23:05, "BlackIce" wrote: > Hi, > > I have some questions regarding performance. > > Lets says I