I wrote my own analyzer: When I view the tokens returned from the
StemFilter the words I'm searching for are returned from the "Token Next()"
function. My terms are under 10,000. I have used luke before how can I make
it use a custom analyzer? Thank you for the information.
.... <general code and default stop words> ....
public final TokenStream tokenStream(String fieldName, Reader
reader) {
TokenStream result = new StandardTokenizer( reader );
result = new LowerCaseFilter( result );
result = new StopFilter( result, stoptable );
result = new ISOLatin1AccentFilter( result );
result = new ItalianStemFilter( result, excltable );
return result;
}
}
-----Original Message-----
From: Erik Hatcher [mailto:[EMAIL PROTECTED]
Sent: Wednesday, November 30, 2005 8:47 AM
To: java-user@lucene.apache.org
Subject: Re: Why are tokens not being indexed?
What Analyzer are you using? Have a look at the Analyzer demo with
Lucene in Action's code, or from my java.net article so you can
analyze your analyzer. Also try out Luke, it really is handy for
seeing inside your index.
And is your text really long (> 10,000 terms)? If so, you'll need
to set the max. field length higher on IndexWriter (I always use
Integer.MAX_VALUE).
Erik
On 30 Nov 2005, at 08:04, Combs, Craig wrote:
> I have a body of text which is being added to a document as
> unstored. All
> the words in the body text are coming through in the token stream for
> analyzing. For some reason I can search on some of tokens and
> others I can
> not.
>
> Take the following string:
>
> "L'amministrazione di Uniface View consente un elevato grado di
> flessibilità
> e può essere realizzata in base ai requisiti del proprio sito. Nel
> manuale
> Guida all'amministrazione di Uniface View vengono descritte le
> procedure di
> amministrazione del sistema. Utilizzare le informazioni riportate
> di seguito
> per creare il portale più adatto alle proprie esigenze."
>
> If I search for "consente" or "elevato" no results are found. If I
> search
> for "view" or "grado" results are found. I find an explanation for
> this
> behavior. These tokens are being returned to the reader but don't
> seem to
> be making it into the index.
>
> Any ideas on how to debugs or an explanation why this might be
> happening?
>
> Regards,
> Craig Combs
>
>
>
>
> The contents of this e-mail are intended for the named addressee
> only. It
> contains information that may be confidential. Unless you are the
> named
> addressee or an authorized designee, you may not copy or use it, or
> disclose
> it to anyone else. If you received it in error please notify us
> immediately
> and then destroy it.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
