David Balmain wrote:
> doc.add_field(:field, "yada yada yada",
> Field::Store::NO, # or YES
> Field::Index::TOKENIZED, # or UNTOKENIZED
> Field::TermVector::YES) # or anything else but NO
I got this far:
------ BEGIN CODE SNIPPET ------
# Read weblog data
weblogs = YAML::load(File.open("weblogs.yml"))
# Walk over weblogs and save all data.
print "--- Analyzing weblogs:\n"
weblogs.each do |weblog, id|
content = ""
print " * Indexing weblog #{weblog}/#{id} "
# Load the appropriate file for parsing.
weblogdata = YAML::load(File.open("./data/#{id}"))
weblogdata[:posts].each do |id, post|
# Clean up content
# by removing all UBB blocks. This will cut-out some content. I
consider this
# loss a plus :D
content = content + "\n\n" +
post[:text].gsub(/\[[^\]]+\][^\[]+\[[^\]]+\]/i, "")
#content.gsub!(/\[[^\]]+\][^\[]+\[[^\]]+\]/i, "")
end
# Create a new document
doc = Document.new
doc.add_field(:id, weblog, Field::Store::YES, Field::Index::TOKENIZED,
Field::TermVector::NO)
doc.add_field(:content, content, Field::Store::NO,
Field::Index::TOKENIZED, Field::TermVector::YES)
# And add to the index.
index << doc
index.flush
print "done.\n"
end
------ END CODE SNIPPET ------
I Index about 23000 weblogs with their weblog id as the document id and
the content by termvector. Now I want to compare two weblogs. So what
you suggest is that I retrieve the term-vectors for both documents and
calculate the dotproduct of the two vectors myself; or is there a nice
Ferret-way to do this?
Thanks in advance,
Jeroen Bulters
--
Posted via http://www.ruby-forum.com/.
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
