On 5/27/06, Jeroen Bulters <[EMAIL PROTECTED]> wrote:
> I Index about 23000 weblogs with their weblog id as the document id and
> the content by termvector. Now I want to compare two weblogs. So what
> you suggest is that I retrieve the term-vectors for both documents and
> calculate the dotproduct of the two vectors myself; or is there a nice
> Ferret-way to do this?
Until now I haven't really used the TermVectors so this probably isn't
the best way to do it but here goes (this is very rough);
def cosine_similarity(index_reader, doc1, doc2)
tv1 = index_reader.get_term_vector(doc1, :data)
terms1 = tv1.terms
freqs1 = tv1.freqs
matrix = {}
terms1.size.times {|i| matrix[terms1[i]] = [freqs1[i], 0]}
tv2 = index_reader.get_term_vector(doc2, :data)
terms2 = tv2.terms
freqs2 = tv2.freqs
terms2.size.times {|i| (matrix[terms2[i]] ||= [0])[1] = freqs2[i]}
dot_product = matrix.values.inject(0) {|dp, (a,b)| dp += a*b}
lengths_product = Math.sqrt(freqs1.inject(0) {|sp, f| sp += f*f} *
freqs2.inject(0) {|sp, f| sp += f*f})
dot_product / lengths_product
end
I'd be interested to hear how you go with this. If performance is poor
I can add something like this to the C code.
Hope this helps,
Dave
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
