[
https://issues.apache.org/jira/browse/LUCENE-8197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393078#comment-16393078
]
Adrien Grand commented on LUCENE-8197:
--------------------------------------
Thank Robert and Dawid for having a look, I folded feedback in:
- Fixed brackets. I've used ]a,b[ for my entire scholarship to mean open
intervals but that seems to be only a thing in France and I'm not fully
converted to parenthesis yet. :)
- I renamed arguments in javadocs, but not in formulas to keep explanations
easy to read: a-> scalingFactor for the log function, k -> pivot for the satu
and sigm functions and a -> exp for the sigm function. If you can think of
better names, I'm open to suggestions.
- Added javadocs for these params
- Made the explanation break down.
- Removed Class.hashCode() usage
I've also been exploring the idea of making it easier to use and added two
utility methods:
- One takes a cutover document frequency and computes an approximation of the
IDF for terms that have such a frequency in field that is searched. This allows
to do something like: if the query term is very specific (freq << x) then the
query-dependent score should dominate the final score as it matters more, but
on the other hand if the query term is very general (freq >> x) then the
feature score should dominate.
- Another one computes the geometric mean (which is the only metric we can
compute with index stats) of the indexed features for usage as the pivot value
in the satu function. I expect it to be good enough to get started.
Both these utility methods combined mean that you can start using the satu
function in a way that shouldn't be too wrong. I added an example how to do it
in the class-level javadocs.
> Make top-k queries fast when static scoring signals are incorporated into the
> score
> -----------------------------------------------------------------------------------
>
> Key: LUCENE-8197
> URL: https://issues.apache.org/jira/browse/LUCENE-8197
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Priority: Minor
> Fix For: master (8.0)
>
> Attachments: LUCENE-8197.patch, LUCENE-8197.patch, LUCENE-8197.patch
>
>
> Block-max WAND (LUCENE-8135) and some earlier issues made Lucene faster at
> computing the top-k matches of boolean queries.
> It is quite frequent that users want to improve ranking and end up scoring
> with a formula that could look like {{bm25_score + w * log(alpha +
> pagerank)}} (w and alpha being constants, and pagerank being a per-document
> field value). You could do this with doc values and {{FunctionScoreQuery}}
> but unfortunately this will remove the ability to optimize top-k queries
> since the scoring formula becomes opaque to Lucene.
> I'd like to add a new field that allows to store such scoring signals as term
> frequencies, and new queries that could produce {{log(alpha + pagerank)}} as
> a score. Then implementing the above formula can be done by boosting this
> query with a boost equal to {{w}} and adding this boosted query as a SHOULD
> clause of a {{BooleanQuery}}. This would give Lucene the ability to compute
> top-k hits faster, especially but not only if the index is sorted by
> decreasing pagerank.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
