David-Sarah Hopwood wrote:
> If you are subclassing CommonTokenStream or TokenRewriteStream,
> I think it should be sufficient to override the LT method as follows:
>
> protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);
>
> @Override public Token LT(int k) {
> Token t = super.LT(k);
> return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
> }
>
> (The EOF_TOKEN doesn't actually exist in the token stream; it is
> returned only when you look ahead using LT.)
>
> However, I haven't tested this,I've tested it now; it appears to work. > and I don't know whether there are any other places where CommonToken is > hardcoded. > >> i'm currently using 3.1.3 (since that's what my eclipse ide >> supports).... is there is workaround for 3.1.3??? > > The workaround I used was: > > - Delete the TokenLabelType option; > > - Change my code so that it no longer assumes that all tokens are > instances of MyToken. Emitted and error tokens will still be of > class MyToken, but fragment and EOF tokens might not be. > (Fragment tokens only occur if you refer to a named child fragment > in a lexer rule. To check that you're not doing this, search for > "new CommonToken" in the generated lexer.) I forgot to say that this is in addition to overriding getMissingSymbol as I described earlier. -- David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
signature.asc
Description: OpenPGP digital signature
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
