I have updated the Highlighter code in CVS to support tokenizers that generate overlapping tokens.
The Junit test rig has a new example test that uses a "SynonymTokenizer" which generates multiple tokens in the same position for the same input token eg (the token "football" is expanded into tokens "soccer","footie" and "football"). The Formatter interface had to be changed to take a new "TokenGroup" object instead of a single token but I doubt any code changes in clients are required because most people use the default Formatter implementation and haven't created their own implementations. Cheers Mark --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
