[
https://issues.apache.org/jira/browse/SOLR-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162196#comment-14162196
]
Mikhail Khludnev commented on SOLR-5444:
----------------------------------------
[~steff1193] expanding docset is reasonable - default constants and behavior do
not work well for your numbers.
As an another workaround, you can try enlarge filter cache, that should avoid
these docset collecting. Do all docsets fit into RAM? Then, enabling docvalues
should bypass those SimpleFacets hotspots.
> Slow response on facet search, lots of facets, asking for few facets in
> response
> --------------------------------------------------------------------------------
>
> Key: SOLR-5444
> URL: https://issues.apache.org/jira/browse/SOLR-5444
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Affects Versions: 4.4
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: docvalue, faceted-search, performance
> Fix For: 4.9, Trunk
>
> Attachments: Profiiling_SimpleFacets_getListedTermCounts_path.png,
> Profiling_SimpleFacets_getTermCounts_path.png,
> Responsetime_func_of_facets_asked_for-Simple_DocSetCollector_fix.png,
> Responsetime_func_of_facets_asked_for.png,
> SOLR-5444_ExpandingIntArray_DocSetCollector_4_4_0.patch,
> SOLR-5444_simple_DocSetCollector_4_4_0.patch
>
>
> h5. Setup
> We have a 6-Solr-node (release 4.4.0) setup with 12 billion "small" documents
> loaded across 3 collections. The documents have the following fields
> * a_dlng_doc_sto (docvalue long)
> * b_dlng_doc_sto (docvalue long)
> * c_dstr_doc_sto (docvalue string)
> * timestamp_lng_ind_sto (indexed long)
> * d_lng_ind_sto (indexed long)
> From schema.xml
> {code}
> <dynamicField name="*_dstr_doc_sto" type="dstring" indexed="false"
> stored="true" required="true" docValues="true"/>
> <dynamicField name="*_lng_ind_sto" type="long" indexed="true"
> stored="true"/>
> <dynamicField name="*_dlng_doc_sto" type="dlng" indexed="false"
> stored="true" required="true" docValues="true"/>
> ...
> <fieldType name="dstring" class="solr.StrField" sortMissingLast="true"
> docValuesFormat="Disk"/>
> <fieldType name="dlng" class="solr.TrieLongField" precisionStep="0"
> positionIncrementGap="0" docValuesFormat="Disk"/>
> {code}
> timestamp_lng_ind_sto decides which collection documents go into
> We execute queries on the following format:
> * q=timestamp_lng_ind_sto:\[x TO y\] AND d_lng_ind_sto:(a OR b OR ... OR n)
> *
> facet=true&facet.field=a_dlng_doc_sto&facet.zeros=false&facet.mincount=1&facet.limit=<asked-for-facets>&rows=0&start=0
> h5. Problem
> We see very slow response-time when hitting large number of rows, spanning
> lots of facets, but only ask for "a few" of those facets
> h5. Concrete example of query to get some concrete numbers to look at
> With x and y plus a, b ... n set to values so that
> * The timestamp_lng_ind_sto:\[x TO y\] part of the search-criteria alone hit
> about 1.7 billion documents (actually all in one (containing 4.5 billion
> docs) of the three collections - but that is not important)
> * The d_lng_ind_sto:(a OR b OR ... OR n) part of the search-criteria alone
> hit about 500000 documents
> * The combined search-criteria (timestamp_lng_ind_sto AND'ed with
> d_lng_ind_sto) hit about 200000 documents
> The following graph shows responsetime as a function of <asked-for-facets>
> (in query)
> !Responsetime_func_of_facets_asked_for.png!
> Note that responsetime is high for "low" <asked-for-facets>, and that it
> increases fast (but linearly) in <asked-for-facets> up until
> <asked-for-facets> is somewhere inbetween 5000 (where responsetime is close
> to 1000 secs) and 10000 (where responsetime is about 5 secs). For values of
> <asked-for-facets> above 10000 responsetime stays "low" at between 1-10 secs
> Looking at the code and profiling it is clear that the change to better
> responsetime occurs when SimpleFacets.getFacetFieldCounts changes from using
> getListedTermCounts to using getTermCounts.
> The following image shows profiling information during a request with
> <asked-for-facets> at about 2000.
> !Profiiling_SimpleFacets_getListedTermCounts_path.png!
> Note that
> * SimpleFacets.getListedTermCounts is used (green box)
> * 91% of the time spent performing the query is spent in
> DocSetCollector-constructor (red box). During this concrete query 125000
> DocSetCollection-objects are created spending 710 secs all in all. Additional
> investigations show that the time is spent allocating huge int-arrays for the
> "scratch"-int-array. Several thousands of those DocSetCollection-constructors
> create int-arrays at size above 1 million - that takes time, and also leaves
> a nice little job of the GC'er afterwards.
> * The actual search-part of the query takes only 0.5% (4 secs) of the
> combined time executing the query (blue box)
> The following image shows profiling information during a request with
> <asked-for-facets> at about 10000
> !Profiling_SimpleFacets_getTermCounts_path.png!
> Note that
> * SimpleFacets.getTermCounts is used (green box)
> * The actual search-part of the query now takes 70% (11 secs) of the combined
> time executing the query (blue box)
> h5. What to do about this?
> * I am not sure why there are two paths that SimpleFacets.getFacetFieldCounts
> can take (getListedTermCounts or getTermCounts) - but I am pretty sure there
> is a good reason. It seems like getListedTermCounts is used when
> <asked-for-facets> is noticeable lower than the total number of facets hit
> (believe it is when <asked-for-facets> * 1.5 + 10 is below actual number of
> facets hit)
> * *One solution* could be to just drop the getListedTermCounts-path and
> always go getTermCounts, but that is probably not at good idea, because
> getListedTermCounts is probably there for a performance reason (in other
> scenarios)
> * The comment above DocSetCollection.scratch says
> {code}
> // in case there aren't that many hits, we may not want a very sparse
> // bit array. Optimistically collect the first few docs in an array
> // in case there are only a few.
> final int[] scratch;
> {code}
> The comment seems reasonable. But when we look at what values are used as
> "smallSetSize" for the DocSetCollection-constructor, it is always "maxDoc >>
> 6" (basically dividing by 64) - this value depends on maxDoc and will be high
> if maxDoc is high. In my case maxDoc is 50+ million a lot of the times
> resulting in "smallSetSize"s of 1+ million (that is not "a few"). I am very
> much in doubt why you want "smallSetSize" to increase as maxDoc increase -
> why not just always a low (fixed or something) value for "smallSetSize"? Is
> it ever a good idea with huge int-arrays for the "scratch"-array?
> * *Another solution* would be to never create "scratch"-arrays with size
> above e.g. 50
> * *There are probably several other potential solutions*
> I would really want your opinion on what solution to make, so that I do not
> unintentionally break good performance-optimizations, just because I missed
> some points explaining why the code is as it is today!?
> *Note* I have filed this as a 4.4 issues, because that is the platform I use
> for my tests etc. But I am sure the problem also exists on 4.5.1 (or whatever
> the latest 4.x release is)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
