David-Sarah Hopwood wrote: > David-Sarah Hopwood wrote: >> If you are subclassing CommonTokenStream or TokenRewriteStream, >> I think it should be sufficient to override the LT method as follows: >> >> protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);
Make this 'public static final' in order for the nextToken() code below
to work.
>> @Override public Token LT(int k) {
>> Token t = super.LT(k);
>> return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
>> }
>>
>> (The EOF_TOKEN doesn't actually exist in the token stream; it is
>> returned only when you look ahead using LT.)
>>
>> However, I haven't tested this,
>
> I've tested it now; it appears to work.
I spoke too soon -- there is another method that needs to be overridden
in the lexer:
@lexer::members {
@Override public Token nextToken() {
Token t = super.nextToken();
return t != Token.EOF_TOKEN ? t : MyTokenStream.MY_EOF_TOKEN;
}
}
This is getting too complicated. Just to ensure that all tokens are of
the overridden type, we have to:
- set the TokenLabelType option
- override emit() and nextToken() in the lexer
- override getMissingSymbol() in the parser
- override LT() in the token stream
- upgrade to ANTLR 3.2
and this is all quite fragile and possibly dependent on the Java target.
It clearly calls for a redesign of the way TokenLabelType is handled.
--
David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
signature.asc
Description: OpenPGP digital signature
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
