On Thu, Jul 30, 2009 at 7:12 PM, <oh...@cox.net> wrote:
> I was wonder if there is a list of special characters for the standard 
> analyzer?
>
> What I mean by "special" is characters that the analyzer considers break 
> characters.
> For example, if I have something like "foo=something", apparently the analyzer
> considers this as two terms, "foo" and "something.

Hi Jim,

This is what I could find in the docs...

StandardAnalyzer uses StandardTokenizer

http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/analysis/standard/StandardTokenizer.html
* Splits words at punctuation characters, removing punctuation.
However, a dot that's not followed by whitespace is considered part of
a token.
* Splits words at hyphens, unless there's a number in the token, in
which case the whole token is interpreted as a product number and is
not split.
* Recognizes email addresses and internet hostnames as one token.

Also, these are the tokens that will be removed..

  public static final String[] ENGLISH_STOP_WORDS = {
    "a", "an", "and", "are", "as", "at", "be", "but", "by",
    "for", "if", "in", "into", "is", "it",
    "no", "not", "of", "on", "or", "such",
    "that", "the", "their", "then", "there", "these",
    "they", "this", "to", "was", "will", "with"
  };

Thanks,
Phil

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to