Ken Williams wrote:
>
>
> On 6/4/10 4:16 PM, "Junkman" <[email protected]> wrote:
>> The way nextToken() is overriden, it first returns the token matched by
>> the rule, and subsequently any additional queued token before matching a
>> new token in the input stream.
>
> Maybe I'm being dense here, but I don't think that's what it's doing:
>
> public Token nextToken() {
> return tokenQueue.isEmpty() ? super.nextToken() : tokenQueue.poll();
> }
>
> If tokenQueue() is non-empty, it always uses it. On the *next* invocation,
> when it's empty, it will call super.nextToken().
>
>
Think of tokens generated by a single rule invocation as a set. The set
is generated in/under "super.nextToken()", AFTER the queue has been
tested to be empty. Among the tokens in the set, the "matching" token
is returned first, because that's what Lexer.nextToken()
("super.nextToken()") returns.
If that's still not clear, I suggest you put the generated lexer under a
debugger (like Jim suggested in another thread ;-) and trace it from
nextToken() - will give you better explanation than my verbiage.
Best regards.
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
