The 'in' in boring is enough for the lexer to decide you are trying to write
interesting and when it gets to the 'g' it finds your lexer rule is in error.
Because there is nothing else to consume such a word, you cannot use a
predicate as ANTLR will see that it does not need the predicate.
If you do this though:
grammar T;
root
: ignored_segment (interesting_segment ignored_segment)*
;
ignored_segment
: ( ~ INTERESTING_KEYWORD )*
;
interesting_segment
: INTERESTING_KEYWORD brace_scope
;
brace_scope
: OPEN_BRACE
(
~( OPEN_BRACE | CLOSE_BRACE )
| brace_scope
)*
CLOSE_BRACE
;
INTERESTING_KEYWORD : 'interesting' ;
WORDS : ('a'..'z')+ {$type = UNMATCHED; } ;
OPEN_BRACE : '{' ;
CLOSE_BRACE : '}' ;
WS
: (' '|'\t'|'\n'|'\r')+
{
$channel=HIDDEN;
};
UNMATCHED : . ;
You will see that it does more checking before selecting the
INTERESTING_KEYWORD rule.
Jim
> -----Original Message-----
> From: [email protected] [mailto:antlr-interest-
> [email protected]] On Behalf Of Michael Coupland
> Sent: Monday, November 30, 2009 3:28 PM
> To: [email protected]
> Subject: [antlr-interest] Over-eager lexer?
>
> Hi,
>
> I'm trying to use ANTLR to extract portions of a text file, and I'm
> having a strange lexer problem. I've boiled my problem down to a
> pretty simple case:
>
> I want to match the interesting segments of C-like input, as
> delineated by the keyword 'interesting' and matching braces:
>
> /* blah blah blah - this part should be ignored */
> interesting { /* this is the part */ { /* that is matched */ }
> }
> /* also ignored */
>
>
> Here's my ANTLR file:
>
> grammar test;
>
> root
> : ignored_segment (interesting_segment ignored_segment)*
> ;
>
> ignored_segment
> : ( ~ INTERESTING_KEYWORD )*
> ;
>
> interesting_segment
> : INTERESTING_KEYWORD brace_scope
> ;
>
> brace_scope
> : OPEN_BRACE (
> ( options {greedy=true;} : ~( OPEN_BRACE | CLOSE_BRACE )
> | brace_scope )
> )* CLOSE_BRACE
> ;
>
> WS
> : (' '|'\t'|'\n'|'\r')+
> {
> $channel=HIDDEN;
> };
>
> INTERESTING_KEYWORD : 'interesting' ;
> OPEN_BRACE : '{' ;
> CLOSE_BRACE : '}' ;
> UNMATCHED : . ;
>
> When I run the grammar on the following input, I get the expected
> behavior.
>
> humdrum
> interesting { xxx }
> humdrum
>
>
> However, running a slightly different input through the ANTLRWorks
> debugger (or C runtime generated code) gives an error:
>
> boring
> interesting { xxx }
> boring
>
>
> I get the following lexer complaint in the debugger output:
>
> line 1:5 mismatched character 'g' expecting 't'
>
>
> It's like the lexer sees the 'in' in 'boring' and then refuses to give
> up trying to match an 'interesting' token. Can someone explain why
> this is happening, and how to solve it? I realize I'm kinda abusing
> the lexer/parser, but the grammar seems like the best way to
> accomplish my goal.
>
> Thanks!
> Michael
>
> PS. I'm vaguely aware of the concept of filter lexers, but I don't
> think I can do the brace matching I need with them? Also, I can't
> meaningfully test them in ANTLRWorks, since it doesn't show the lexer
> results. I rely on ANTLRWorks heavily to author my grammars before
> running them in the ANTLR C Runtime.
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
