Re: SolrCloud query logs, change from 4.10 to 5.5

[Erick Erickson]

Been there, done that

You might be glad to know that there are a couple of tickets to reduce
the verbosity of the logs (or, more accurately, move some of the
logging to DEBUG level and allow a switch at startup) that should make
staring at logs less of a chore..

One other signal that a query is a sub-query to a shard is if
"distrib=false" is set but I don't know how reliable that would be for
your purpose...

Best,
Erick

On Fri, Sep 23, 2016 at 7:29 AM, Elaine Cario  wrote:
> Oh, never mind.  Apparently staring at logs has led to blindness...I do see
> the "master" query with the full elapsed time and hit count, and indeed,
> there is a  parameter "_" with some tracking number which links all the
> queries together.
>
>
> On Thu, Sep 22, 2016 at 7:32 PM, Elaine Cario  wrote:
>
>> We're in the process of upgrading from SolrCloud 4.10 to 5.5, and we
>> noticed a change in how distributed queries get logged.
>>
>> In Solr 4.10 we noted that the original node receiving the query logged
>> the query with a full hit count and elapsed time for the entire query,
>> using the original request handler (we don't use the default /select
>> handler).  The other nodes logged the queries sent out from the original
>> node, using the /select request handler.  These entries just included the
>> query stats from that particular node/shard.
>>
>> This made it easy when log-diving to differentiate between the stats for
>> the entire query completion, vs the individual stats for each shard, and we
>> were also able to detect any unexpected network latencies between the
>> shards.
>>
>> But now we are finding in Solr 5.5 that each shard just logs its own
>> stats, using the original request handler and there's no log entry for the
>> query as a whole.  This is making some of our existing log analysis
>> difficult when we try to tie it back to our other application logs.
>>
>> So, I have 2 questions:
>>
>> - is there a way to force a log entry for the complete query?
>> - is there some definitive way to link together all the log entries for a
>> query across the shards, e.g. some query parameter placed there by Solr?
>>  (In some cases our applications due add a custom param with a transaction
>> ID, but it's not consistent and I wonder if Solr is doing something or can
>> be configured to add something)
>>
>> Thanks.
>>

Re: SolrCloud query logs, change from 4.10 to 5.5

[Elaine Cario]

Oh, never mind.  Apparently staring at logs has led to blindness...I do see
the "master" query with the full elapsed time and hit count, and indeed,
there is a  parameter "_" with some tracking number which links all the
queries together.


On Thu, Sep 22, 2016 at 7:32 PM, Elaine Cario  wrote:

> We're in the process of upgrading from SolrCloud 4.10 to 5.5, and we
> noticed a change in how distributed queries get logged.
>
> In Solr 4.10 we noted that the original node receiving the query logged
> the query with a full hit count and elapsed time for the entire query,
> using the original request handler (we don't use the default /select
> handler).  The other nodes logged the queries sent out from the original
> node, using the /select request handler.  These entries just included the
> query stats from that particular node/shard.
>
> This made it easy when log-diving to differentiate between the stats for
> the entire query completion, vs the individual stats for each shard, and we
> were also able to detect any unexpected network latencies between the
> shards.
>
> But now we are finding in Solr 5.5 that each shard just logs its own
> stats, using the original request handler and there's no log entry for the
> query as a whole.  This is making some of our existing log analysis
> difficult when we try to tie it back to our other application logs.
>
> So, I have 2 questions:
>
> - is there a way to force a log entry for the complete query?
> - is there some definitive way to link together all the log entries for a
> query across the shards, e.g. some query parameter placed there by Solr?
>  (In some cases our applications due add a custom param with a transaction
> ID, but it's not consistent and I wonder if Solr is doing something or can
> be configured to add something)
>
> Thanks.
>

SolrCloud query logs, change from 4.10 to 5.5

[Elaine Cario]

We're in the process of upgrading from SolrCloud 4.10 to 5.5, and we
noticed a change in how distributed queries get logged.

In Solr 4.10 we noted that the original node receiving the query logged the
query with a full hit count and elapsed time for the entire query, using
the original request handler (we don't use the default /select handler).
The other nodes logged the queries sent out from the original node, using
the /select request handler.  These entries just included the query stats
from that particular node/shard.

This made it easy when log-diving to differentiate between the stats for
the entire query completion, vs the individual stats for each shard, and we
were also able to detect any unexpected network latencies between the
shards.

But now we are finding in Solr 5.5 that each shard just logs its own stats,
using the original request handler and there's no log entry for the query
as a whole.  This is making some of our existing log analysis difficult
when we try to tie it back to our other application logs.

So, I have 2 questions:

- is there a way to force a log entry for the complete query?
- is there some definitive way to link together all the log entries for a
query across the shards, e.g. some query parameter placed there by Solr?
 (In some cases our applications due add a custom param with a transaction
ID, but it's not consistent and I wonder if Solr is doing something or can
be configured to add something)

Thanks.