Hi Mikhail, you are my hero! I added your patch to 4.10.4, recompiled, tested and pushed from testing stage to online system. What a difference!
Right after restart the performance increase for faceting is times 10. Qtime for MatchAllDocsQuery(*:*) and docValues and faceting went down from around 35 seconds to 3.5 seconds for faceting. After 1 hour under load the qtime is somewhere around 15 percent performance increase for faceting. This patch is a must have! Regards Bernd Am 07.08.2015 um 08:45 schrieb Bernd Fehling: > > > Am 06.08.2015 um 17:48 schrieb Mikhail Khludnev: >> On Thu, Aug 6, 2015 at 3:56 PM, Bernd Fehling < >> bernd.fehl...@uni-bielefeld.de> wrote: >> >>> >>> >>> Am 06.08.2015 um 14:33 schrieb Upayavira: >>>> Typically such performance issues with faceting are to do with the time >>>> spend uninverting the index before calculating the facet counts. >>>> >>>> If you indexed the fields with docValues enabled, perhaps you could then >>>> use them for faceting, which might improve performance. >>> >>> Well, this is against my observations. When I used uninverted fields >>> without >>> docValues I had a much better 99 percentile qtime but a very high heap >>> consumption. >>> Now with docValues the heap usage went down, but the 99 percentile >>> qtime for MatchAllDocsQuery(*:*) went up to above 33 seconds. >>> >> >> Note about performance optimization for DocValues faceting in forthcoming >> 5.3 >> > > Thanks for pointing this out. > Do you think that it is possible to merge this patch into 4.10.4 or > is 5.3 to far away from 4.10.4 in this area of code? > > >> >>> >>>> >>>> If you are using a non-docValues field, and the second query is faster, >>>> then you could add the query to your static warming, look for >>>> newSearcher in your solrconfig.xml. That will execute your query, >>>> warming the caches used by faceting, before a new searcher is made >>>> available for searches. >>> >>> The q=*.* with sorting and facetting is always the first query I'm doing >>> at static warming and it helped until switching to docValues :-( >>> >>> Bernd >>> >>>> >>>> Upayavira >>>> >>>> On Thu, Aug 6, 2015, at 12:38 PM, Toke Eskildsen wrote: >>>>> On Thu, 2015-08-06 at 13:00 +0200, Bernd Fehling wrote: >>>>>> Single Index Solr 4.10.4, optimized Index, 76M docs, 235GB index size. >>>>>> >>>>>> I was analysing my solr logs and it turned out that I have some queries >>>>>> which are above 30 seconds qtime while normally the qtime is below 1 >>> second. >>>>>> Looking closer about the queries it turned out that this is for >>> MatchAllDocsQuery(*:*). >>>>>> Next was to turn debugQuery on and see where the bottleneck is. >>>>>> The result was that the facetting is consuming most of the qtime. >>>>>> >>>>>> So the question is, are facets or is facetting not cached? >>>>> >>>>> As far as I know it is not. 35 seconds for a match-all faceting sounds >>>>> fairly on par with what we are seeing (250M docs, 900GB shard). >>>>> >>>>> Of course response time is very depending on the field itself. If you >>>>> have very few unique values in your facet field(s), you might try >>>>> facet.method=enum. If that is not the case, your best bet would probably >>>>> be to cache the match-all outside of Solr. >>>>> >>>>>> My assumption is that the queryResultCache is catching such a >>> MatchAllDocsQuery(*:*). >>>>> >>>>> It only stores the docIDs. >>>>> >>>>> I don't know why there is is no all_parameters -> complete_response >>>>> cache in Solr. >>>>> >>>>> - Toke Eskildsen, State and University Library, Denmark >>>>> >>>>> >>> > -- ************************************************************* Bernd Fehling Bielefeld University Library Dipl.-Inform. (FH) LibTec - Library Technology Universitätsstr. 25 and Knowledge Management 33615 Bielefeld Tel. +49 521 106-4060 bernd.fehling(at)uni-bielefeld.de BASE - Bielefeld Academic Search Engine - www.base-search.net *************************************************************