In the attachment, you may find my implementation. I
use the second way (I have no idea which way is better
without testing against performance, precision, recall
etc). i.e.,
"blah C1C2C3 blah C4 blah" -> "blah", "C1", "C1C2"+0, "C2", "C2C3"+0, "C3", "blah",
"C4", "blah" (+0 to indicate a positionIncrement of zero, rather
than 1.)
This looks great. I will commit it soon. (I'm currently on vacation, and very far behind on my email.)
Did you connect this to LuceneDocumentAnalysis and the QueryTranslator, so that it is used when indexing and searching? If you do this, please send the patch files.
By the way, is there any way in lucene API which can
tell you the occurrences of a Term in the whole
collection, not just the "idf" value. I try to
implement a language model (probabilistic) based
similarity measure.
The only way to find this in Lucene is to iterate through the TermDocs, summing the freq(). That is not optimal. It would not be hard to extend Lucene to store this number as well.
Doug
-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - digital self defense, top technical experts, no vendor pitches, unmatched networking opportunities. Visit www.blackhat.com
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
