Hello,

I am using ANTLR 3.2.

With a lexer grammar such as this one:

        lexer grammar Z;
        options {
            filter=true;
        }
        IS_KEYWORD                : 'IS' {skip();};

The skip() is ignored and the token gets inserted in the token stream.

If I change to filter=false, skip is honored.

The problem might come from the nextToken() method which seems to be
overriden in the generated lexer code when filter=true.

The nextToken() method in org.antlr.runtime.Lexer has code like this:

        try {
            mTokens();
            if ( state.token==null ) {
                emit();
            }
            else if ( state.token==Token.SKIP_TOKEN ) {
                continue;
            }
            return state.token;
        }

While the generated equivalent code (in Z.java) looks like this:

        try {
            int m = input.mark();
            state.backtracking=1; 
            state.failed=false;
            mTokens();
            state.backtracking=0;
            
            if ( state.failed ) {
                input.rewind(m);
                input.consume(); 
            }
            else {
                emit();
                return state.token;
            }
        }
So whatever the state.token value, the token is emitted.

Or I messed up and there some other explanation?

Thanks for your help.

Fady


List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: 
http://www.antlr.org/mailman/options/antlr-interest/your-email-address

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"il-antlr-interest" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to 
[email protected]
For more options, visit this group at 
http://groups.google.com/group/il-antlr-interest?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to