Dave you might want to connect JVisualVm and see if there's any pattern with latency and garbage collection. That's a frequent culprit for periodic hits in latency.
More info here http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/jmx_connections.html There's a couple GC implementations in Java that can be tuned as needed With JvisualVM You can also add the mbeans plugin to get a ton of performance stats out of Solr that might help debug latency issues. Doug Sent from my Windows Phone From: Dave Seltzer Sent: 11/21/2013 8:42 PM To: solr-user@lucene.apache.org Subject: Re: Periodic Slowness on Solr Cloud Lots of questions. Okay. In digging a little deeper and looking at the config I see that <nrtMode>true</nrtMode> is commented out. I believe this is the default setting. So I don't know if NRT is enabled or not. Maybe just a red herring. I don't know what Garbage Collector we're using. In this test I'm running Solr 4.5.1 using Jetty from the example directory. The CPU on the 8 nodes all stay around 70% use during the test. The nodes have 28GB of RAM. Java is using about 6GB and the rest is being used by OS cache. To perform the test we're running 200 concurrent threads in JMeter. The threads hit HAProxy which loadbalances the requests among the nodes. Each query is for a random word out of a list of about 10,000 words. Some of the queries have faceting turned on. Because we're heavily loading the system the queries are returning quite slowly. For a simple search, the average response time was 300ms. The peak response time was 11,000ms. The spikes in latency seem to occur about every 2.5 minutes. I haven't spent that much time messing with SolrConfig, so most of the settings are the out-of-the-box defaults. Where should I start to look? Thanks so much! -Dave On Thu, Nov 21, 2013 at 6:53 PM, Mark Miller <markrmil...@gmail.com> wrote: > Yes, more details… > > Solr version, which garbage collector, how does heap usage look, cpu, etc. > > - Mark > > On Nov 21, 2013, at 6:46 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > > > How real time is NRT? In particular, what are you commit settings? > > > > And can you characterize "periodic slowness"? Queries that usually > > take 500ms not tail 10s? Or 1s? How often? How are you measuring? > > > > Details matter, a lot... > > > > Best, > > Erick > > > > > > > > > > On Thu, Nov 21, 2013 at 6:03 PM, Dave Seltzer <dselt...@tveyes.com> > wrote: > > > >> I'm doing some performance testing against an 8-node Solr cloud cluster, > >> and I'm noticing some periodic slowness. > >> > >> > >> http://farm4.staticflickr.com/3668/10985410633_23e26c7681_o.png > >> > >> I'm doing random test searches against an Alias Collection made up of > four > >> smaller (monthly) collections. Like this: > >> > >> MasterCollection > >> |- Collection201308 > >> |- Collection201309 > >> |- Collection201310 > >> |- Collection201311 > >> > >> The last collection is constantly updated. New documents are being > added at > >> the rate of about 3 documents per second. > >> > >> I believe the slowness may due be to NRT, but I'm not sure. How should I > >> investigate this? > >> > >> If the slowness is related to NRT, how can I alleviate the issue without > >> disabling NRT? > >> > >> Thanks Much! > >> > >> -Dave > >>
