I was running on Windows, and when I moved to linux the problem disappeared (I'm assuming because linux automatically uses cferret?). Thanks for the help!
Justin
On 6/16/06, David Balmain <
[EMAIL PROTECTED]> wrote:
On 6/17/06, Justin Kan < [EMAIL PROTECTED]> wrote:
> Hi,
>
> I'm using the StandardAnalyzer to build an index, and passing in Documents
> that have Fields that contain large tokens (22+ characters) interpersed with
> normal English words. This seems to cause the IndexWriter to slow to a
> crawl. Is this a known issue, or am I doing something wrong?
Hi Justin,
I haven't come accross this problem? Are you on Windows by any chance?
Currently Ferret is just generally slow on Windows because it is pure
Ruby code. One problem large tokens may cause is the general increase
in the number of terms in the index which can slow down indexing a
little but it would surprise me if it was making a huge difference
unless there was a particularly large number of them.
> If this is a known issue I don't have any problem just not indexing tokens
> longer than a certain length, but what's the best way to eliminate them?
> Using a TokenFilter on my own Analyzer? Sorry for the newbish questions, I'm
> new to ferret having never used lucene. Thanks in advance,
Yes, using a token filter will do the job. Have a look in the analysis
module of Ferret for some examples. I'd be interested to hear if it
makes any difference.
Cheers,
Dave
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
_______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
