We're not using ELB and I have no idea which connector I'm using - I'm
guessing whatever is default (I'm a total noob). This is from my server.xml:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="60000"
URIEncoding="UTF-8" redirectPort="8443" />
--
Nate Fox
Sr Systems Engineer
o: 310.658.5775
m: 714.248.5350
Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
Facebook<http://www.facebook.com/neogov>
NEOGOV <http://www.neogov.com/> is among the top fastest growing software
companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta <
michael.della.bi...@appinions.com> wrote:
> Nate,
>
> We just cleared up a problem similar to this by ditching Elastic Load
> Balancer and switching over to the APR connector in Tomcat. Are you
> using either of those?
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
> <otis.gospodne...@gmail.com> wrote:
> > Hi Nate,
> >
> > Try adding some warmup queries and making sure the setting for using
> > the cold searcher in solrconfig.xml is set to false. Your warmup
> > queries should use facets and sorting if your normal queries use them.
> > In SPM you'll actually see how much time warming up takes, so you'll
> > get a better idea of the "cost" of that (when you don't do it).
> >
> > Otis
> > --
> > Solr & ElasticSearch Support
> > http://sematext.com/
> >
> >
> >
> >
> >
> > On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <n...@neogov.com> wrote:
> >> I was wondering if the warmup stuff was one of the culprits (we dont
> have
> >> warmup's at all - the configs are pretty stock).
> >> As for the system, it seems capable of quite a bit more: memory usage is
> >> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb)
> and
> >> load below 1.00.
> >>
> >> The seed data and queries were put together by one of our developers.
> I've
> >> put all the solrmeter files here:
> >> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
> >> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely
> sure
> >> which file does which specifically.
> >>
> >> Does the system's reaction to a 'fast load' without a warmup sound
> normal?
> >> I would have expected the first couple hundred queries to be very slow
> >> (>500ms) and then the system catch up after a while. But it just dies
> very
> >> quickly and never recovers.
> >>
> >> I'll check out your SPM - I've seen it mentioned before. Thanks!
> >>
> >>
> >>
> >> --
> >> Nate Fox
> >> Sr Systems Engineer
> >>
> >> o: 310.658.5775
> >> m: 714.248.5350
> >>
> >> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> >> Facebook<http://www.facebook.com/neogov>
> >>
> >> NEOGOV <http://www.neogov.com/> is among the top fastest growing
> software
> >> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
> >> the LA Business Journal. We are hiring!<
> http://www.neogov.com/#/company/careers>
> >>
> >>
> >>
> >> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
> >> otis.gospodne...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> In short, certain data structures need to load from index in the
> >>> beginning, (for sorting and faceting) caches need to warm up, JVM
> >>> needs to warm up, etc., so going slowly in the beginning makes sense.
> >>> Why things die after that is a different Q. Maybe it OOMs? Maybe
> >>> queries are very complex? What do your queries look like? I see
> >>> newrelic.jar in the command-line. May want to try SPM for Solr, it
> >>> has better Solr metrics.
> >>>
> >>> Otis
> >>> --
> >>> Solr & ElasticSearch Support
> >>> http://sematext.com/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <n...@neogov.com> wrote:
> >>> > I'm new to solr and I'm load testing our setup to see what we can
> handle.
> >>> > I'm using solrmeter and my problem is a bit odd:
> >>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
> >>> > hundred queries and then tomcat will stop responding completely to
> >>> requests
> >>> > (even though according to lsof -i it is still listening and the java
> >>> > process is still running).
> >>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can
> stop
> >>> > solrmeter after a couple of minutes at that pace and then run at
> >>> 4000/min
> >>> > without issue.
> >>> >
> >>> > It's as if it needs a ramp up time? Also, I noticed (regardless of
> ramp
> >>> up)
> >>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the
> same
> >>> as
> >>> > if I were to run 4k/min without the ramp up. Of note, only the shard
> that
> >>> > solrmeter is pointed to stops responding. The other shard hums along
> >>> > without incident.
> >>> >
> >>> > Setup (everything in AWS):
> >>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
> >>> > (open-jdk-7-headless) : Ubuntu 12.04
> >>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
> >>> > I have ~30k documents in each node (~300Mb on each node)
> >>> >
> >>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
> >>> > packages/solr's example dir. Here's the configs and the end of the
> >>> > catalina.out file:
> https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
> >>> >
> >>> > My main question is two fold:
> >>> > 1. Is this normal behavior for tomcat (to just stop responding
> >>> completely)
> >>> > when it gets overwhelmed? And the only option is to restart it? I
> guess I
> >>> > dont know what it looks like when tomcat/solr cant keep up.
> >>> > 2. Why does it handle better when I give it a lower number of
> queries and
> >>> > then ramp it up? It concerns me that if I have to restart a server
> in the
> >>> > cluster and it gets thrown into the pool of machines that things will
> >>> blow
> >>> > up.
> >>> >
> >>> > As an aside, does this seem like a normal amount of queries (~4k/min)
> >>> that
> >>> > this kind of environment should be able to handle?
> >>>
>
