> I see, so this is intended for use in cases where the a token is
> replaced with a sequence of tokens that represents, as a sequence, a single
> meaning
> that is similar to that of the original token.
You got it.
> Another case, which does not seem to be supported
> is when a token is replaced with a sequence of tokens, each
> representing an *alternative* meaning. Here is an example:
>
> 'dog' -> 'dog', 'pet'
> 'cat' -> 'cat', 'pet'
> 'pet' -> 'pet'
While I see how it would be cool to do this, I think this is not
really what the tokenization interface is for. I could be wrong; lets
see if Doug says anything.
_______________________________________________
Lucene-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/lucene-dev
