On 7/30/2015 3:53 AM, Manohar Sripada wrote:
> We have Solr Cloud (version 4.7.2) setup on 64 shards spread across VMs.
> I see my queries to Solr taking exactly 50 sec intermittently (as
> someone said so :P). This happens once in 10 queries.
> I have enabled log level to TRACE on all the solr nodes. I didn't find
> any issue with the query time on any given shard (max QTime observed on
> a shard is 10 ms). We ran all the tests related to network and
> everything looks fine there.
>
> Whenever the query took 50 sec, I am seeing the below log statements
> for org.eclipse.jetty component. Is this some issue with Jetty?? I could
> see this logs being printed every 11 seconds(/2015-07-24
> 07:06:00, //2015-07-24 07:06:11, ...)/ for 4 times. Attached the
> complete logs during that duration. Can someone please help me here??
<snip>
> /INFO - 2015-07-24 07:06:00.128;
> org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null
> path=/admin/info/logging
> params={_=1437736005493&since=1437734905469&wt=json} status=0 QTime=0
Those logs appear to be caused by someone watching the Logging tab in
the admin UI. This admin UI page refreshes every ten seconds. No queries
are happening during the log you included, only the requests for logging
info. These requests are normally very fast, and in your log, they show
a qtime of zero milliseconds.
64 shards is quite a bit, and as soon as someone talks about a very
large install on virtual machines that is having performance problems, I
suspect that they probably do not have enough resources (memory in
particular) for what they are asking the system to do.
Now it's time for some light reading:
http://wiki.apache.org/solr/SolrPerformanceProblems
Next there are questions. These first bunch of questions are about the
virtual machines themselves, not the host hardware for the virtual machines.
Are you using the jetty (start.jar) included with Solr, or have you
installed Solr into a different jetty?
On the dashboard of the admin UI, in the JVM section, there is an "Args"
parameter, which may have multiple lines. What all is there?
If you add up all the shard replicas on a single virtual machine, how
many docs are there and how much disk space is used by the index data?
Include all replicas in those numbers, even if they duplicate data
that's on another virtual machine.
How much memory does the virtual machine have, and how much of that
memory is allocated to the java heap?
Are all of the virtual machines similar as far as memory config and how
much Solr data they contain?
If you are using a virtual machine platform that you host yourself, then
I need to know how many of these virtual machines are loaded onto each
physical machine, and how much memory that physical machine has. If
you're using AWS, then this question is irrelevant. The allocation of
CPU resources might be important, but it's not as important as memory.
Thanks,
Shawn
