I'd write a TokenFilter for that... much easier. -Yonik Now hiring -- http://tinyurl.com/7m67g
On 9/27/05, Lorenzo Viscanti <[EMAIL PROTECTED]> wrote: > > Hi, I'm trying to modify the StandardTokenizer, in order to get to get a > good tokenization for my needs. > Basically I would like to separate two tokens when I find an apostrophe. I > think I should modify the StandardTokenizer.jj file to do that, but I'm in > trouble while changing the grammar. Can someone help me with the right > syntax? > Thanks, > Lorenzo >
