Hi Mikhail, you are my hero!

I added your patch to 4.10.4, recompiled, tested and pushed from testing stage
to online system. What a difference!

Right after restart the performance increase for faceting is times 10.
Qtime for MatchAllDocsQuery(*:*) and docValues and faceting went down from
around 35 seconds to 3.5 seconds for faceting.
After 1 hour under load the qtime is somewhere around 15 percent performance
increase for faceting.

This patch is a must have!

Regards
Bernd


Am 07.08.2015 um 08:45 schrieb Bernd Fehling:
> 
> 
> Am 06.08.2015 um 17:48 schrieb Mikhail Khludnev:
>> On Thu, Aug 6, 2015 at 3:56 PM, Bernd Fehling <
>> bernd.fehl...@uni-bielefeld.de> wrote:
>>
>>>
>>>
>>> Am 06.08.2015 um 14:33 schrieb Upayavira:
>>>> Typically such performance issues with faceting are to do with the time
>>>> spend uninverting the index before calculating the facet counts.
>>>>
>>>> If you indexed the fields with docValues enabled, perhaps you could then
>>>> use them for faceting, which might improve performance.
>>>
>>> Well, this is against my observations. When I used uninverted fields
>>> without
>>> docValues I had a much better 99 percentile qtime but a very high heap
>>> consumption.
>>> Now with docValues the heap usage went down, but the 99 percentile
>>> qtime for MatchAllDocsQuery(*:*) went up to above 33 seconds.
>>>
>>
>>  Note about performance optimization for DocValues faceting in forthcoming
>> 5.3
>>
> 
> Thanks for pointing this out.
> Do you think that it is possible to merge this patch into 4.10.4 or
> is 5.3 to far away from 4.10.4 in this area of code?
> 
> 
>>
>>>
>>>>
>>>> If you are using a non-docValues field, and the second query is faster,
>>>> then you could add the query to your static warming, look for
>>>> newSearcher in your solrconfig.xml. That will execute your query,
>>>> warming the caches used by faceting, before a new searcher is made
>>>> available for searches.
>>>
>>> The q=*.* with sorting and facetting is always the first query I'm doing
>>> at static warming and it helped until switching to docValues :-(
>>>
>>> Bernd
>>>
>>>>
>>>> Upayavira
>>>>
>>>> On Thu, Aug 6, 2015, at 12:38 PM, Toke Eskildsen wrote:
>>>>> On Thu, 2015-08-06 at 13:00 +0200, Bernd Fehling wrote:
>>>>>> Single Index Solr 4.10.4, optimized Index, 76M docs, 235GB index size.
>>>>>>
>>>>>> I was analysing my solr logs and it turned out that I have some queries
>>>>>> which are above 30 seconds qtime while normally the qtime is below 1
>>> second.
>>>>>> Looking closer about the queries it turned out that this is for
>>> MatchAllDocsQuery(*:*).
>>>>>> Next was to turn debugQuery on and see where the bottleneck is.
>>>>>> The result was that the facetting is consuming most of the qtime.
>>>>>>
>>>>>> So the question is, are facets or is facetting not cached?
>>>>>
>>>>> As far as I know it is not. 35 seconds for a match-all faceting sounds
>>>>> fairly on par with what we are seeing (250M docs, 900GB shard).
>>>>>
>>>>> Of course response time is very depending on the field itself. If you
>>>>> have very few unique values in your facet field(s), you might try
>>>>> facet.method=enum. If that is not the case, your best bet would probably
>>>>> be to cache the match-all outside of Solr.
>>>>>
>>>>>> My assumption is that the queryResultCache is catching such a
>>> MatchAllDocsQuery(*:*).
>>>>>
>>>>> It only stores the docIDs.
>>>>>
>>>>> I don't know why there is is no all_parameters -> complete_response
>>>>> cache in Solr.
>>>>>
>>>>> - Toke Eskildsen, State and University Library, Denmark
>>>>>
>>>>>
>>>
> 

-- 
*************************************************************
Bernd Fehling                    Bielefeld University Library
Dipl.-Inform. (FH)                LibTec - Library Technology
Universitätsstr. 25                  and Knowledge Management
33615 Bielefeld
Tel. +49 521 106-4060       bernd.fehling(at)uni-bielefeld.de

BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************

Reply via email to