I don't know the behaviour of the Japanese Analyzer you are using.
Can you add to your example diagnosis the Token.getPositionIncrement,
Token.startOffset and Token.endOffset for each of the tokens?
The highlighter groups tokens with overlapping start and end offsets
into a single TokenGroup for the purposes of highlighting. This allows
TokenStreams which produce multiple synonyms for the same source token
to work. This behaviour was also required to get the CJKAnalyzer to
work. It could be that the Analyzer you are using is producing a stream
of tokens which *all* overlap?
Cheers
Mark
___________________________________________________________
To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
