OK so I solved the question about the query that returns no results and
still takes time - I needed to add the facet.mincount=1 parameter and this
reduced the time to 200-300 ms instead of seconds.
I still could't figure out why a query that returns very few results (like
query number 2) still takes seconds to return even with
the facet.mincount=1 parameter.
I couldn't understand why the facet pivot takes so much time on 299 docs.
Does anyone have any idea?
Example Query:
----------------------------------------------------
(2)
q=*:*&fq=(trimTime:[2012-09-04T15:23:48Z TO *])&fq=(Severity:("High"
"Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
*])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High")) OR
(Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
NumFound: 299
Times(ms):
Qtime: 2,756 Query: 307 Facet: 2,449
On Thu, Sep 20, 2012 at 5:24 PM, Yuval Dotan <yuvaldo...@gmail.com> wrote:
> Hi,
>
> We have a system that inserts logs continuously (real-time).
> We have been using the Solr facet pivot feature for querying and have been
> experiencing slow query times and we were hoping to gain some insights with
> your help.
> schema and solrconfig are attached
>
> Here are our questions (data below):
>
> 1. Why is facet time so long in (3) and (5) - in cases where there are
> 0 or very few results?
> 2. We ran two queries that are only differ in the time limit (for the
> second query - time range is very small) - we got the same time for both
> queries although the second one returned very few results - again why is
> that?
> 3. Is there a way to improve pivot facet time?
>
> System Data:
>
> Index size: 63 GB
> RAM:4Gb
> CPU: 2 x Xeon E5410 2.33GHz
> Num of Documents: 109,278,476
>
>
> query examples:
>
> ---------------------------------------------------------------------------------------------------------------------------------------------
> (1)
> Query:
> q=*:*&fq=(trimTime:[2012-09-04T14:29:24Z TO
> *])&fq=(trimTime:[2012-09-04T14:29:24Z TO
> *])&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound:
> 11,407,889
>
> Times (ms):
> Qtime: 3,239 Query: 353 Facet: 2,885
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> (2)
> Query:
> q=*:*&fq=(trimTime:[2012-09-04T15:23:48Z TO *])&fq=(Severity:("High"
> "Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
> *])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High")) OR
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound: 299
>
> Times(ms):
> Qtime: 2,756 Query: 307 Facet: 2,449
>
> ---------------------------------------------------------------------------------------------------------------------------------------------
> (3)
> Query:
> q=*:*&fq=(trimTime:[2012-09-11T12:55:00Z TO *])&fq=(Severity:("High"
> "Critical"))&fq=(trimTime:[2012-09-04T15:23:48Z TO
> *])&fq=(Confidence_Level:("N/A")) OR (Confidence_Level:("Medium-High")) OR
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound: 7
>
> Times(ms):
> Qtime: 2,798 Query: 312 Facet: 2,485
>
> ---------------------------------------------------------------------------------------------------------------------------------------------
> (4)
> Query:
> q=*:*&fq=(trimTime:[2012-09-04T15:43:16Z TO
> *])&fq=(trimTime:[2012-09-04T15:43:16Z TO *])&fq=(product:("Application
> Control")) OR (product:("URL
> Filtering"))&f.appi_name.facet.sort=index&f.appi_name.facet.limit=-1&f.app_risk.facet.sort=index&f.app_risk.facet.limit=-1&f.matched_category.facet.sort=index&f.matched_category.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.appi_name.facet.method=enum&facet.pivot=appi_name,app_risk,matched_category,trimTimeex&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound: more than 30M
>
> Times(ms): Qtime: 23,288
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> (5)
> Query:
> q=*:*&fq=(trimTime:[2012-09-05T06:03:55Z TO *])&fq=(Severity:("High"
> "Critical"))&fq=(trimTime:[2012-09-05T06:03:55Z TO *])&fq=(product:("IPS"))
> OR (product:("SmartDefense"))&fq=(action:("Detect")) OR
> (action:("mixed"))&fq=(Confidence_Level:("Medium-High")) OR
> (Confidence_Level:("High"))&f.product.facet.sort=index&f.product.facet.limit=-1&f.Severity.facet.sort=index&f.Severity.facet.limit=-1&f.trimTime.facet.sort=index&f.trimTime.facet.limit=-1&facet=true&f.product.facet.method=enum&facet.pivot=product,Severity,trimTime
>
> NumFound: 0
>
> Times(ms): Qtime: 2,971 Query: 218 Facet: 2,753
>
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
