Hi Adrien,
I thought about merging the two into one value that I could use in the
scoring function but failed to find a way to do so.
The scoring function is:
log(1+(max(docrFreq-delta,0)) / (delta * d_u* p(w\c) ) ) + log( delta *
d_u / |d|)
(also included as image bellow)
{\log{{\left({1}+\frac{{\max{\left({d}{o}{c}{F}{r}{e}{q}-\delta,{0}\right)}}}{{\delta\cdot{d}_{u}\cdot{p}{\left({w}{c}\right)}}}\right)}}}+{\log{{\left(\delta\cdot\frac{{d}_{u}}{{{\left|{d}\right|}}}\right)}}}
Where:
d_u is the unique term count
|d| is the document length.
p(w\c) is a language model computed on the corpus that gives the
probability that the corpus generates the word w
delta is a parameter (which value is between 0 and 1).
My problem here is that I need the unique term count once alone and once
divided by the document length, so I can't really craft a single value
combining the two that i can use in the scoring function.
I am trying to work around the problem by thinking about how i can
change the function while keeping the idea of what the different terms
express but couldn't find a satisfying solution yet.
At least I know i should not search how to create a doc value anymore ! :)
Thanks a lot for your help !
Romaric
Le 21/05/2019 à 22:23, Adrien Grand a écrit :
Hi Romaric,
Indeed similarities are not expected to create doc value fields, they
should only populate norms. The similarity API has been changed in 8.0
and similarities no longer have access to the reader context, they are
now expected to work with only term frequency and a length
normalization factor as per-document contributions to the score.
One challenge is that Lucene now mandates that scores do not increase
when the norm value increases, which makes it hard to record both the
unique term count and the total term count in the norm. What does the
scoring function look like, maybe there are ways that we could record
both the unique term count and total term count in a single long
depending on how the scoring formula merges them (maybe not!).
On Tue, May 21, 2019 at 3:28 PM Romaric Pighetti
<romaric.pighe...@francelabs.com> wrote:
Hi,
Thanks Adrien for the quick and accurate answer.
Digging into the implementation I saw that the document length is
already stored there and as I need both the unique term count and the
length, I can't just replace one with the other.
The Similarity class documentation states that it is possible to store
additional values using NumericDocValuesField that could be accessed at
query time using a LeafReader.
From my understanding, using the LeafReaderContext when building the
SimScorer should allow me to get access to the NumericDocValuesField.
The problem is I don't get how to create and store a new
NumericDocValuesField from the Similarity. My guess is that it should
happen within the computeNorm function again as it is the only function
called at indexing time. However I am unable to understand how to create
and store this information from that function.
If you have any advice that would be really helpful.
Thanks.
Romaric
Le 20/05/2019 à 12:16, Adrien Grand a écrit :
Hi Romaric,
You are right, computeNorm is the right place to compute and record
the number of unique terms of a document. Your computeNorm function
would look something like this:
@Override
public final long computeNorm(FieldInvertState state) {
return SmallFloat.intToByte4(state.getUniqueTermCount());
}
And then in your scorer you could convert the norm back to the unique
term count by doing SmallFloat.byte4ToInt on it. The SmallFloat
methods are useful to encode this count on one byte, which trades some
accuracy but is usually the right trade-off.
On Mon, May 20, 2019 at 12:05 PM Romaric Pighetti
<romaric.pighe...@francelabs.com> wrote:
Hi,
I am currently implementing a new similarity class into lucene which is
based on a language model with absolute discount.
I am basing my work on the work already done in the
LMDirichletSimilarity and LMJelinekMercerSimilarity which are really close.
However to end my implementation I need to get the number of unique
terms present in the document, and this information seems to be
unavailable natively from within the score function.
The computeNorm function which is in the Similarity class seems to be
the right place to compute (or read) and store this statistic but I am
not sure.
So I am reaching you to know if I am on the right track and if you have
any advice on how I could access this statistic from the computeNorm
function if possible ?
I would like the implementation to be as clean as possible with regards
to Lucene's code expectation to be able to submit it for integration
once it is done.
Thanks for your help,
Regards.
--
Romaric Pighetti
R&D - FranceLabs
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
--
Romaric Pighetti
R&D - FranceLabs
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
--
Romaric Pighetti
R&D - FranceLabs
