Re: Question about fieldNorm

[Brendan Grainger]

Hi Yonik,

I just realized that the stemmer does make a difference because of  
synonyms. So on indexing using the new stemmer "converter hanger  
assembly replacement" gets expanded to: "converter hanger assembly  
assemble replacement" so there are 5 terms which gets a length norm of  
0.4472136 instead of 0.5. Still unsure how it gets 0.4375 though as  
the result for the field norm though unless I have a boost of 0.9783  
somewhere there.

Brendan


On Jun 11, 2008, at 1:37 PM, Yonik Seeley wrote:

That is strange... did you re-index or change the index?  If so, you
might want to verify that docid=3454 still corresponds to the same
document you queried earlier.

-Yonik


On Wed, Jun 11, 2008 at 1:09 PM, Brendan Grainger
<[EMAIL PROTECTED]> wrote:
I've just changed the stemming algorithm slightly and am running a  
few tests
against the old stemmer versus the new stemmer. I did a query for  
'hanger'
and using the old stemmer I get the following scoring for a  
document with
the title: Converter Hanger Assembly Replacement

6.4242806 = (MATCH) sum of:
2.5697122 = (MATCH) max of:
  0.2439919 = (MATCH) weight(markup_t:hanger in 3454), product of:
    0.1963516 = queryWeight(markup_t:hanger), product of:
      6.5593724 = idf(docFreq=6375, numDocs=1655591)
      0.02993451 = queryNorm
    1.2426275 = (MATCH) fieldWeight(markup_t:hanger in 3454),  
product of:
      1.7320508 = tf(termFreq(markup_t:hanger)=3)
      6.5593724 = idf(docFreq=6375, numDocs=1655591)
      0.109375 = fieldNorm(field=markup_t, doc=3454)
  2.5697122 = (MATCH) weight(title_t:hanger^2.0 in 3454), product of:
    0.5547002 = queryWeight(title_t:hanger^2.0), product of:
      2.0 = boost
      9.265229 = idf(docFreq=425, numDocs=1655591)
      0.02993451 = queryNorm
    4.6326146 = (MATCH) fieldWeight(title_t:hanger in 3454),  
product of:
      1.0 = tf(termFreq(title_t:hanger)=1)
      9.265229 = idf(docFreq=425, numDocs=1655591)
      0.5 = fieldNorm(field=title_t, doc=3454)
3.8545685 = (MATCH) max of:
  0.12199595 = (MATCH) weight(markup_t:hanger^0.5 in 3454), product  
of:
    0.0981758 = queryWeight(markup_t:hanger^0.5), product of:
      0.5 = boost
      6.5593724 = idf(docFreq=6375, numDocs=1655591)
      0.02993451 = queryNorm
    1.2426275 = (MATCH) fieldWeight(markup_t:hanger in 3454),  
product of:
      1.7320508 = tf(termFreq(markup_t:hanger)=3)
      6.5593724 = idf(docFreq=6375, numDocs=1655591)
      0.109375 = fieldNorm(field=markup_t, doc=3454)
  3.8545685 = (MATCH) weight(title_t:hanger^3.0 in 3454), product of:
    0.8320503 = queryWeight(title_t:hanger^3.0), product of:
      3.0 = boost
      9.265229 = idf(docFreq=425, numDocs=1655591)
      0.02993451 = queryNorm
    4.6326146 = (MATCH) fieldWeight(title_t:hanger in 3454),  
product of:
      1.0 = tf(termFreq(title_t:hanger)=1)
      9.265229 = idf(docFreq=425, numDocs=1655591)
      0.5 = fieldNorm(field=title_t, doc=3454)

Using the new stemmer I get:

5.621245 = (MATCH) sum of:
2.248498 = (MATCH) max of:
  0.24399184 = (MATCH) weight(markup_t:hanger in 3454), product of:
    0.19635157 = queryWeight(markup_t:hanger), product of:
      6.559371 = idf(docFreq=6375, numDocs=1655589)
      0.029934512 = queryNorm
    1.2426274 = (MATCH) fieldWeight(markup_t:hanger in 3454),  
product of:
      1.7320508 = tf(termFreq(markup_t:hanger)=3)
      6.559371 = idf(docFreq=6375, numDocs=1655589)
      0.109375 = fieldNorm(field=markup_t, doc=3454)
  2.248498 = (MATCH) weight(title_t:hanger^2.0 in 3454), product of:
    0.5547002 = queryWeight(title_t:hanger^2.0), product of:
      2.0 = boost
      9.265228 = idf(docFreq=425, numDocs=1655589)
      0.029934512 = queryNorm
    4.0535374 = (MATCH) fieldWeight(title_t:hanger in 3454),  
product of:
      1.0 = tf(termFreq(title_t:hanger)=1)
      9.265228 = idf(docFreq=425, numDocs=1655589)
      0.4375 = fieldNorm(field=title_t, doc=3454)
3.372747 = (MATCH) max of:
  0.12199592 = (MATCH) weight(markup_t:hanger^0.5 in 3454), product  
of:
    0.09817579 = queryWeight(markup_t:hanger^0.5), product of:
      0.5 = boost
      6.559371 = idf(docFreq=6375, numDocs=1655589)
      0.029934512 = queryNorm
    1.2426274 = (MATCH) fieldWeight(markup_t:hanger in 3454),  
product of:
      1.7320508 = tf(termFreq(markup_t:hanger)=3)
      6.559371 = idf(docFreq=6375, numDocs=1655589)
      0.109375 = fieldNorm(field=markup_t, doc=3454)
  3.372747 = (MATCH) weight(title_t:hanger^3.0 in 3454), product of:
    0.83205026 = queryWeight(title_t:hanger^3.0), product of:
      3.0 = boost
      9.265228 = idf(docFreq=425, numDocs=1655589)
      0.029934512 = queryNorm
    4.0535374 = (MATCH) fieldWeight(title_t:hanger in 3454),  
product of:
      1.0 = tf(termFreq(title_t:hanger)=1)
      9.265228 = idf(docFreq=425, numDocs=1655589)
      0.4375 = fieldNorm(field=title_t, doc=3454)

The thing that is perplexing is that the fieldNorm for the title_t  
field is
different in each of the explanations, ie: the fieldNorm using the  
old
stemmer is: 0.5 = fieldNorm(field=title_t, doc=3454). For the new  
stemmer
0.4375 = fieldNorm(field=title_t, doc=3454). I ran the title  
through both
stemmers and get the same number of tokens produced. I do no index  
time
boosting on the title_t field. I am using DefaultSimilarity in both
instances. So I figured the calculated fieldNorm would be:

field boost * lengthNorm = 1 * 1/sqrt(4) = 0.5

I wouldn't have thought that changing the stemmer would have any  
impact on
the fieldNorm in this case. Any insight? Please kick me over to the  
lucene
list if you feel this isn't appropriate here.

Regards
Brendan