On Thu, 2012-06-07 at 10:01 +0200, Andrew Laird wrote:
For our needs we don't really need to know that a particular facet has
exactly 14,203,527 matches - just knowing that there are more than a
million is enough. If I could somehow limit the hit counts to a
million (say) [...]
It should be feasible to stop the collector after 1M documents has been
processed. If nothing else then just by ignoring subsequent IDs.
However, the ID's received would be in index-order, which normally means
old-to-new. If the nature of the corpus, and thereby the facet values,
changes over time, this change would not be reflected in the facets that
has many hits as the collector never reaches the newer documents.
it seems like that could decrease the work required to
compute the values (just stop counting after the limit is reached) and
potentially improve faceted search time - especially when we have 20-30
fields to facet on. Has anyone else tried to do something like this?
The current Solr facet implementation treats every facet structure
individually. It works fine in a lot of areas but it also means that the
list of IDs for matching documents is iterated once for every facet: In
the sample case, 14M+ hits * 25 fields = 350M+ hits processed.
I have been experimenting with an alternative approach (SOLR-2412) that
packs the terms in the facets as a single structure underneath the hood,
which means only 14M+ hits processed in the current case. Unfortunately
it is not mature and only works for text fields.
- Toke Eskildsen, State and University Library, Denmark
