It seems that I missed one scenario. Now if I give the input as ANTLR ANTLR then I get an error.
It seems that the second token is a super-set of the first token and this case both the words are treated as ANTLRTOKEN . A similar situation is described in the section "Ambiguities and Non determinisms" section of book "The definitive ANTLR guide", that talks about the ambiguities in lexer rules, but I am not sure how to resolve this. How can this be done without making the first rule also as SPECIAL_STRING ? In general, does any section of the book define any guidelines as to what rules should be lexer rules and what should be parser rules ? Thanks! Regards, Achint On Wed, Aug 12, 2009 at 11:07 PM, Achint Mehta <[email protected]>wrote: > Hi Benoit, > > The solution suggested by you solved the problem. > > Many thanks. > > Regards, > Achint > > > On Wed, Aug 12, 2009 at 2:10 AM, Benoit Fouletier > <[email protected]<benblo%[email protected]> > > wrote: > >> I think special_string should be a lexer rule, not a parser rule: rename >> it to SPECIAL_STRING. Also, the lexer depends on the order with which you >> define tokens, so make sure you put ANTLRTOKEN above SPECIAL_STRING. >> >> >> On Wed, Aug 12, 2009 at 7:34 AM, Achint Mehta <[email protected]>wrote: >> >>> >>> Hi, >>> >>> I am stuck at a seemingly trivial problem. >>> I have written a simplified sample grammar which has this issue. >>> >>> In the grammar I have a rule to extract a generic string >>> special_string: (CHAR | '=' | '.' | '-' | '@' )+ ; >>> >>> and a token ANTLR which is defined as: >>> ANTLRTOKEN:'ANTLR'; >>> >>> A rule which parses two words ( the first of which has to be ANTLR is >>> defined as follows): >>> requestline : ANTLRTOKEN WHITESPACE special_string ; >>> >>> It seems that if the input word begins with keyword "ANTLR", then that >>> word is treated as it is beginning with ANTLRTOKEN and passed to the parser. >>> i.e. an input text "ANTLR ANTLRWORKS", loosely seems to be treated as >>> sequence of >>> ANTLRTOKEN WHITESPACE ANTLRTOKEN special_string >>> >>> The whole grammar file is as follows: (This grammer simply parses any >>> word followed by the keyword ANTLR) >>> >>> ----------------------------------------------------------------------------- >>> grammar sample_parser; >>> >>> options >>> { >>> language=C; >>> } >>> >>> requestline : ANTLRTOKEN WHITESPACE special_string ; >>> special_string: (CHAR | '=' | '.' | '-' | '@' )+ ; >>> >>> WHITESPACE : ( '\t' | ' ' | '\u000C' )+; >>> NEWLINE: ('\r')? '\n'; >>> CHAR: (('a'..'z')|('A'..'Z')); >>> ANTLRTOKEN:'ANTLR'; >>> >>> ----------------------------------------------------------------------------- >>> >>> If i provide the input as >>> ANTLR WORKS >>> >>> Then everything works fine and I don't get any error. >>> >>> Now if I provide the input as >>> ANTLR ANTLRWORKS >>> Then I get the error as >>> >>> ---------------------------------------------------------------------------- >>> input(1) : error 5 : Unexpected token, at offset 5 >>> near [Index: 2 (Start: 24666934-Stop: 24666938) ='ANTLR', type<4> >>> Line: 1 LinePos:5] >>> : missing elements... >>> >>> ---------------------------------------------------------------------------- >>> >>> Seems that the lexer treats the sub-string ANTLR in ANTLRWORKS as the >>> token ANTLRTOKEN and passes it to parser which is not expecting that token. >>> >>> Is there a way to tell antlr to not to break the input word ANTLRWORKS >>> into token and treat the whole word as special_string ? >>> >>> Can somebody help me getting around this issue ? >>> >>> Thanks in advance. >>> >>> Also, I am using the following version of the library, etc. >>> java version "1.6.0_14" >>> ANTLR version 3.1.3 >>> Target language : C >>> C runtime library version: 3.1.3 >>> gcc compiler: 4.3.3 (Ubuntu 4.3.3-5ubuntu4) >>> >>> >>> Regards, >>> Achint >>> >>> >>> List: http://www.antlr.org/mailman/listinfo/antlr-interest >>> Unsubscribe: >>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address >>> >>> >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
