ANTLR makes no assumptions about whitespace; ANTLR lexers just slice up a stream (array) of characters into tokens according to a lexer grammar. For most programming language grammars, whitespace can be ignored (and lexer grammars tend to put whitespace tokens on a "hidden" channel) since it basically serves to separate meaningful tokens.
If you want to require or forbid whitespace in parser rules, you either have to not ignore whitespace or insert semantic predicates to look for whitespace or its absence. This gets messy fast, so modern languages avoid the old FORTRAN "interesting" lexical behavior, and you will not find much expertise in dealing with really outre lexer problems here--it is unusual to find a need to solve such problems. --Loring ----- Original Message ---- > From: Graham Wideman <[email protected]> > To: [email protected] > Sent: Wednesday, August 5, 2009 5:10:18 PM > Subject: [antlr-interest] Whitespace: More than meets the eye? > > Hi folks: > > Could someone clue me in to the zen of whitespace (or where this is discussed > in > The Books or online)? By that I mean as follows: > > I see that grammars generally have a lexer rule that specifies the pattern > for > whitespace, but what's the correspondence between that and what ANTLR does > with > lexer rules, where the WHITESPACE token is generally not used. > > I'm assuming that ANTLR makes assumptions that whitespace either may or must > appear between tokens that appear in lexer rules, right? Either that or it's > picking up something from lexer rules that's subtler than I've noticed. > > To pick a simple example, suppose you're defining what a function call should > look like: > > functionCall > : Identifier LParen args RParen -> .... > > ... what variations does one need in order to assert that whitespace is, or > is > not, permitted or required between Identifier and LParen? > > Thanks! > > Graham > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
