Hi Paul,
Thanks for your detailed reply! It really helped alot.
However, I am experiancing some conflicts.
For one of the documents in result set, when i use
IndexReader fir=FilterIndexReader.open(index);
byte[] fNorm=fir.norm(Body);
System.out.println(FNorm: + fNorm[306]);
Document d=fir.document(306);
Field f=d.getField(Body);
System.out.println(Body: + f.stringValue());
This gives me out fNorm 113, whereas total number of term (including
stop-words) are 42 in this particular field of selected document. In the
explanation , fieldNorm (field=Body, doc=306) is 0.1562, which is approx
41 term words for that field in that documents. So explanation values
makes sense with real data, while including all stop words like to,it,
the etc.
So, my question is,
Am i getting the norm values from right place?
Is there any way to find out number of indexed terms for each
document?
Please advise!
Thanks,
Zia
On Wed, 2004-09-29 at 08:17, Paul Elschot wrote:
Zia,
On Tuesday 28 September 2004 21:22, you wrote:
Hi,
I'm trying to learn the Scoring mechanism of Lucene. I want to fetch
each parameter value individually as they are collectively dumped out by
Explanation. I've managed to pull out TF and IDF values using
DefaultSimilarity and FilterIndexReader, but not sure from where to get
the fieldNorm and queryNorm from.
The norms are here:
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/index/IndexReader.html#norms(java.lang.String)
The resulting array is indexed by the document number for the IndexReader.
With the default similarity, each norm is the inverse square root of the number of
indexed terms in the
document field. However, there are only 8 bits available to encode this value, so
it's quite rough.
The default queryNorm is here:
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/DefaultSimilarity.html#queryNorm(float)
There is an explanation of the scoring in the javadocs of Similarity.
There has been some discussion on an idf factor that was missing from this
documentation,
I don't know whether the docs have been adapted for this.
Also is there any reference about how normalisation has been
implemented?
See above, DefaultSimilarity is the default implementation of the Similarity
interface.
queryNorm() takes a sumOfSquaredWeights, where the weights are the term weights
from the query. It returns the square root.
It may be that the sum of squared weights should be a sum of square rooted weights
and that queryNorm should return a square then.
I posted this on lucene-user on 20 September:
http://issues.apache.org/eyebrowse/[EMAIL PROTECTED]msgNo=10023
It's only a normalisation, so it doesn't affect the order of the search results much.
Taking the square roots of the query term weights would have
the query weights directly apllied to the the query term density in the document
field,
whereas now the weights seem to be applied to the square root of the density.
The density value is an approximation, see above for the rough field norms.
Regards,
Paul Elschot
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Zia Syed [EMAIL PROTECTED]
Smartweb Research Center, Robert Gordon University
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
