Erik Hatcher wrote:

> Rather than changing StandardAnalyzer, you could create a custom  
> Analyzer that is something along the lines of StandardTokenizer  ->  
> custom apostrophe splitting filter -> ISOLatinFilter. 

Why do not include that in the FrenchStemFilter "next()" method itself ?
It will be a bad design ?

And I'm quite concerned with performance issue, but it seem's to me that
your solution will only affect "APOSTROPHE" typed token, so the overhead
will be unexistant, right ?

> You get a special type for words with interior apostrophes from 
> StandardTokenizer (look at StandardFilter to see how that works). You
> could create a simple TokenFilter that splits apostrophe'd tokens 
> into two.

I'm not sure to figure out to do that efficiently. Is it something like
that ? :

<code>

private Stack subTokens; //previously initialized

public final Token next() throws IOException {
  Token t = null;
  if (subTokens != null && !subTokens.empty) {
    t = subTokens.pop();
  } else {
    t = input.next();
    if (t != null)
    {
      String type = t.type();
      if (type == APOSTROPHE_TYPE) {
        tokenizeApostrophe(t, subTokens);
      }
    }
  }
  return t;
}

</code>

with "tokenizeApostrophe(Token, Stack)" that split on conditions the
token into 2 others, and push them on the stack.

> Maybe it's simple enough also to expand "j" and "l" into "je" and
> "le" in the same step too?

It will be simple, but I'm not sure yet I want to expand them back.
Maybe it will be useful to index the "j" token after all.

Anyway thanks for your quick answer,

-- 
Hugo

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Reply via email to