Try this to get you started:
-----------------------------------------------------
@lexer::members {
// Queue to hold additional tokens
private java.util.Queue<Token> tokenQueue = new
java.util.LinkedList<Token>();
// Include queue in reset().
public void reset() {
super.reset();
tokenQueue.clear();
}
// Queued tokens are returned before matching a new token.
public Token nextToken() {
if (tokenQueue.peek() != null)
return tokenQueue.poll();
return super.nextToken();
}
}
MATCHED_TOKEN: ...
{
// Add additional tokens to the queue.
tokenQueue( new CommonToken( ... ) );
}
-----------------------------------------------------
MATCHED_TOKEN is returned first, and additional tokens queued by
MATCHED_TOKEN's action are returned subsequently before matching new
tokens in the input stream.
Instantiate the additional token accordingly if you need input stream
context - see Lexer.emit().
Ken Williams wrote:
>
> On 6/3/10 4:18 PM, "Jim Idle" <[email protected]> wrote:
>
>> Add to an array or collection then get nextToken to remove from the
>> collection. It si slower to do this so it isn't the default way.
>
> Yeah, that's what the book says. =)
>
> It seems like there are some subtleties involved, though - there's a lot of
> bookkeeping in nextToken() that looks kind of scary (e.g. the
> current-line-number stuff, the default-channel stuff, etc.), and if I
> override it I'm really not confident I'll do it correctly. I'm also unsure
> how mTokens(), emit(), and nextToken() cooperate with their member
> variables.
>
> I tried this simple-minded implementation, and started getting out-of-bounds
> exceptions:
>
> @lexer::members {
> List<Token> tokBuf = new ArrayList<Token>();
> public Token nextToken() {
> while (tokBuf.isEmpty()) {
> emit();
> }
> return tokBuf.remove(0);
> }
> public void emit(Token token) {
> tokBuf.add(token);
> }
> }
>
>
> So if someone does have a working example, I'd love to see it!
>
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
