Thanks for your answers. Yes, of COURSE it works as you say. Somehow, after not really worrying about how the lexer works, my brain got stuck thinking that lexer and parser work more analogously than they actually do.
Whereas at any juncture the parser only tries certain rules as predicted by the grammar and the current state, the lexer effectively "tries all its rules" every time it's starting to discern the next token.
So in that process, if the next characters match a rule that discards the characters, (a la whitespace), then that pattern functions as an optional separator.
And I also see that in order to do anything with whitespace at the parser level, either whitespace has to not be discarded (in which case many parser rules will have to deal with it) or custom code will need to be included in the relevant rules to look at the hidden channel etc.
OK, I think this is clear to me now. Back to the regularly scheduled programming...
Thanks again,
Graham
At 8/5/2009 07:38 PM, Gerald Rosenberg wrote:
At 07:00 PM 8/5/2009, Graham Wideman wrote:
Hi Loring
Thanks for your comments. So when you say that "ANTLR makes no assumptions", I take that to mean ANTLR makes these assumptions:
No assumptions means exactly that. ANTLR itself has no concept of whitespace. An input text is just a sequence of tokens and it entirely up to the lexer rules to group the tokens in some meaningful fashion. ANTLR does have a concept of completely tossing (or hiding) tokens if, once recognized, they convey no further meaning in recognizing the input text. That is, the only information they carry is that they separate other groups of tokens.
ANTLR allows any token to be skipped or hidden -- no presumption is made about the type or value.
So|if|your||input|text|was|||structured|in}some|strange|way.
The so-called white space rule could be:
BARS: ( '|' | '}' | '.' ) {$channel=HIDDEN;} ;
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "il-antlr-interest" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [email protected]
For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en
-~----------~----~----~----~------~----~------~--~---
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
