[
https://issues.apache.org/jira/browse/LUCENE-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754470#action_12754470
]
Mark Miller edited comment on LUCENE-1908 at 9/11/09 10:37 PM:
---------------------------------------------------------------
Looks great!
bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing
performance considerstions (?).
Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)|
normalization factor with our document length factor?
That's how it appears to me anyway -
for |V(d)| you have many options right?
the cosine normalization - your standard euclidean length - |V(d)|
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene
uses by default
or Okapi's formula,
or ...
So we are replacing (which everyone generally does) not dropping right?
And I don't think we are replacing for performance reasons (though it is
complicated to calculate) - we are replacing because its not a great
normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer
documents will still have higher tf's and more distinct terms - so it
unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics).
So its not generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and
short preferred too much)
Again, I'm not an IR guy, but thats what my modest take is.
* edit: I suppose you could argue that you could do cosine normalization and
then a further normalization approach on that, and in that sense we are
dropping the cosine normalization because its too expensive. But from what I
can see, it appears more the case that you try and use a normalization factor
that can just replace cosine normalization - like the pivoted normalization
which rotates the cosine normalization curve. I think pivoted is something like
1/root(stuff, ie unique terms), so our norm of 1/root(L) is of a similar,
simpler, vein. So thats why I think we are not really dropping it - we are
choosing one of the variety of replacements.
was (Author: markrmil...@gmail.com):
Looks great!
bq. Document Euclidean norm |V(d)| is excluded by Lucene, for indexing
performance considerstions (?).
Hmm - I'm not sure if that is right either. Are we not replacing the |V(d)|
normalization factor with our document length factor?
That's how it appears to me anyway -
for |V(d)| you have many options right?
the cosine normalization - your standard euclidean length - |V(d)|
or none (eg 1)
or pivoted normalized doc length
or SweetSpotSimilarity's formula
or the quick,dirty,easy, not great default doc length normalization that Lucene
uses by default
or Okapi's formula,
or ...
So we are replacing (which everyone generally does) not dropping right?
And I don't think we are replacing for performance reasons (though it is
complicated to calculate) - we are replacing because its not a great
normalization factor.
Using |V(d)| eliminates info on the length of the orig document - but longer
documents will still have higher tf's and more distinct terms - so it
unnaturally gives them
an advantage (most long docs will be repeated pieces or cover multiple topics).
So its not generally a good normalization factor, and we have a chosen another?
(the one we have chosen isnt great either - long docs are punished too much and
short preferred too much)
Again, I'm not an IR guy, but thats what my modest take is.
> Similarity javadocs for scoring function to relate more tightly to scoring
> models in effect
> -------------------------------------------------------------------------------------------
>
> Key: LUCENE-1908
> URL: https://issues.apache.org/jira/browse/LUCENE-1908
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Doron Cohen
> Assignee: Doron Cohen
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1908.patch
>
>
> See discussion in the related issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
