[ https://issues.apache.org/jira/browse/LUCENE-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16769698#comment-16769698 ]
Mike Sokolov commented on LUCENE-8681: -------------------------------------- There are a bunch of different ways to provide for opt-in here. The most focused would be to just require users to call {{IndexSearcher.search(Query, CollectorManager)}}. That's currently the only way to invoke concurrent collection, We could provide a convenient {{CollectorManager}} via a static method in {{TopFieldCollector}}. I think probably that's enough for this issue? I thought about how to make this easier for users by pushing up to higher-level APIs. There's not an obvious right way, but here's my 2c. Following the current API one would add yet more overrides of {{IndexSearcher.search}} and {{IndexSearcher.searchAfter}} providing the ability to supply a threshold or boolean to enable this feature. I see that there is no such convenience available for {{trackTotalHits}}, and I suspect folks felt there were simply too many overrides already? It certainly seems that way to me. When I see an API getting a great many parameters and overloads with different sets of them, I want to introduce a class to hold them (we don't have optional args and default args in Java). IndexSearcher could take a SearchConfig object that would just be a simple struct holding its various options (sort, numHits, doDocScores, doMaxScore, trackTotalHits, proratedEarlyTerminationThreshold, etc. That would make the search()/searchAfter() methods have simpler signatures (eventually). Having an object class to hold options (like IndexWriterConfig) gives a nice centralized way to provide documentation. Also, in a search UI one often varies pagination and sorting via a different code path than the core Query, so it feels natural to me to use a different abstraction to track those things. > Prorated early termination > -------------------------- > > Key: LUCENE-8681 > URL: https://issues.apache.org/jira/browse/LUCENE-8681 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search > Reporter: Mike Sokolov > Priority: Major > > In this issue we'll exploit the distribution of top K documents among > segments to extract performance gains when using early termination. The basic > idea is we do not need to collect K documents from every segment and then > merge. Rather we can collect a number of documents that is proportional to > the segment's size plus an error bound derived from the combinatorics seen as > a (multinomial) probability distribution. > https://github.com/apache/lucene-solr/pull/564 has the proposed change. > [~rcmuir] pointed out on the mailing list that this patch confounds two > settings: (1) whether to collect all hits, ensuring correct hit counts, and > (2) whether to guarantee that the top K hits are precisely the top K. > The current patch treats this as the same thing. It takes the position that > if the user says it's OK to have approximate counts, then it's also OK to > introduce some small chance of ranking error; occasionally some of the top K > we return may draw from the top K + epsilon. > Instead we could provide some additional knobs to the user. Currently the > public API is {{TopFieldCOllector.create(Sort, int, FieldDoc, int > threshold)}}. The threshold parameter controls when to apply early > termination; it allows the collector to terminate once the given number of > documents have been collected. > Instead of using the same threshold to control leaf-level early termination, > we could provide an additional leaf-level parameter. For example, this could > be a scale factor on the error bound, eg a number of standard deviations to > apply. The patch uses 3, but a much more conservative bound would be 4 or > even 5. With these values, some speedup would still result, but with a much > lower level of ranking errors. A value of MAX_INT would ensure no leaf-level > termination would ever occur. > We could also hide the precise numerical bound and offer users a three-way > enum (EXACT, APPROXIMATE_COUNT, APPROXIMATE_RANK) that controls whether to > apply this optimization, using some predetermined error bound. > I posted the patch without any user-level tuning since I think the user has > already indicated a preference for speed over precision by specifying a > finite (global) threshold, but if we want to provide finer control, these two > options seem to make the most sense to me. Providing access to the number of > standard deviation to allow from the expected distribution gives the user the > finest control, but it could be hard to explain its proper use. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org