To build ST v4's insanely complicated lexer, I couldn't get around an
on-demand (unbuffered) token stream. Note that I don't have an
unbuffered char stream yet but given LookaheadStream<T>, it should be
just as easy. Code for token stream is tiny; see below. It seems to
work so far.
Ter
add //depot/code/antlr/main/runtime/Java/src/main/java/org/antlr/
runtime/UnbufferedTokenStream.java#1
edit //depot/code/antlr/main/runtime/Java/src/main/java/org/antlr/
runtime/misc/LookaheadStream.java#4
edit //depot/code/antlr/main/tool/CHANGES.txt#18
Change 6294 submitted.
package org.antlr.runtime;
import org.antlr.runtime.misc.LookaheadStream;
import java.util.Map;
import java.util.Set;
public class UnbufferedTokenStream extends LookaheadStream<Token>
implements TokenStream {
protected TokenSource tokenSource;
protected int tokenIndex = 0;
public UnbufferedTokenStream(TokenSource tokenSource) {
super(Token.EOF_TOKEN);
this.tokenSource = tokenSource;
}
public Token nextElement() {
Token t = tokenSource.nextToken();
t.setTokenIndex(tokenIndex++);
return t;
}
public TokenSource getTokenSource() { return tokenSource; }
public String toString(int start, int stop) { return "n/a"; }
public String toString(Token start, Token stop) { return "n/a"; }
public int LA(int i) { return LT(i).getType(); }
public String getSourceName() { return tokenSource.getSourceName();
}
}
_______________________________________________
antlr-dev mailing list
[email protected]
http://www.antlr.org/mailman/listinfo/antlr-dev
