Ferret 0.9.3
Ruby 1.8.2
NOT storing file contents in the index.
Only indexing first 25k of each file.
Very large data set (1 million files, 350 Gb)
Code based on snippet from David Balmain's forum posts.
After 6 hours, Ferret bails out with Ruby "exceeds max file size".
Cache:
-rw-r--r-- 1 bill bill 2147483647 2006-06-01 22:45 _ntc6.tmp
-rw-r--r-- 1 bill bill 1690862924 2006-06-01 22:42 _ntc6.prx
-rw-r--r-- 1 bill bill 646302802 2006-06-01 22:42 _ntc6.frq
-rw-r--r-- 1 bill bill 165561698 2006-06-01 22:42 _ntc6.tis
-rw-r--r-- 1 bill bill 50541430 2006-06-01 22:14 _ntc6.fdt
-rw-r--r-- 1 bill bill 8000000 2006-06-01 22:14 _ntc6.fdx
-rw-r--r-- 1 bill bill 2097842 2006-06-01 22:42 _ntc6.tii
-rw-r--r-- 1 bill bill 1000000 2006-06-01 22:42 _ntc6.f0
-rw-r--r-- 1 bill bill 1000000 2006-06-01 22:42 _ntc6.f1
-rw-r--r-- 1 bill bill 30 2006-06-01 22:42 segments
-rw-r--r-- 1 bill bill 16 2006-06-01 22:14 _ntc6.fnm
Code:
#------------
index = Index::Index.new(:path => "/var/cache/ferrets")
max_file_length = 25000
Dir.glob(allfiles).each do
|file|
doc = Document::Document.new()
doc << Document::Field.new(:file, file,
Document::Field::Store::YES,
Document::Field::Index::UNTOKENIZED)
doc << Document::Field.new(:content, IO.read(file, max_file_length),
Document::Field::Store::NO,
Document::Field::Index::TOKENIZED)
index << doc
end
#------------
Is there a workaround, or is this exceeding Ferret's limits?
Thanks! By the way, retrieval is usably fast for my purposes, even on a
big index like this. Very impressive.
--
Posted via http://www.ruby-forum.com/.
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
