Hi, Vic, Unfortunately BM25 uses IDF as well so splitting documents across machines will also affect it. How about storing these as global statistical data for sharing the search on these machines?
The equation of BM25 is clearly stated in Robertson's paper "Simple, proven approaches to text retrieval" (http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-356.pdf) as follows. CW (i,j) = [ CFW (i) * TF (i,j) * (K1+1) ] /[ K1 * ( (1-b) + (b * (NDL (j)) ) ) + TF (i,j) ] CFW(i) is collection frequency weight of term i, TF(i,j) is term frequency of term i, NDL(j) is the normalized document length of document j, and K1 and b are tuning constants. The details are in the paper. Univ. of Amsterdam has provided a downloadable version of a language modelling version of Lucene. Their language model is not BM25 but is quite similar in nature. The version is at: http://ilps.science.uva.nl/Resources/#lm-lucen I have worked on their version a bit, they have created new classes: TermQueryLanguageModel, TermScorerLanguageModel, IndexSearcherLanguageModel, LanguageModelIndexReader etc. I think their work can be a basis. Jianhan -----Original Message----- From: Vic Bancroft [mailto:[EMAIL PROTECTED] Sent: 17 October 2006 13:44 To: java-dev@lucene.apache.org; [EMAIL PROTECTED] Subject: Re: Include BM25 in Lucene? J.Zhu wrote: >If I would like to contribute, what should I do? I am not a good Java >developer myself though. Can I work with someone also interested? > > In some of my group's usage of lucene over large document collections, we have split the documents across several machines. This has lead to a concern of whether the inverse document frequency was appropriate, since the score seems to be dependant on the partioning of documents over indexing hosts. We have not formulated an experiment to determine if it seriously effects our results, though it has been discussed. If someone could elaborate how BM25 or some DFR algorithm would differ from what (TF/IDF) is implemented in lucene, I would be willing to help translate that into java as an indexing/searching option . . . more, l8r, v -- "The future is here. It's just not evenly distributed yet." -- William Gibson, quoted by Whitfield Diffie --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
