[il-antlr-interest: 23749] Re: [antlr-interest] Lexer matching non-matching rule

Sat, 16 May 2009 19:02:21 -0700

Micha wrote: 
On Saturday 16 May 2009 12:52:13 Jesper Larsson wrote:
  
On Sat, 2009-05-16 at 08:27 +0530, Indhu Bharathi wrote:
    
This is because on seeing 'f' of foo lexer has two options - 1. IDENT
2. URL. And it takes the second options since that seems to be longer
that the first alternative. Note that the lexer always tries to match
the longest token possible.
      

but it fails to do so here.
It isn't as cut and dried as the longest match - it is trying to make sense of your ambiguity basically, so when you don't get what you want, you have to guide it with more information. But, it isn't context sensitive so remember that the lexer runs first, then the parser runs with the results. 
  
I can understand the motivation of this restriction in the interest of
keeping the lexer target code at a certain complexity level, but I have
not seen it stated in the documentation. It would have been nice if the
lexer generator had issued a warning.
    

right. I don't like that behavior too, maybe it's because the antlr lexer is 
similar to a recursive descent parser. I think it should just return the 
longest match.
Then it would not be an LL recognizer but just a pattern matcher like lex. You can always use a different lexer, but in practice once you get the hang of where you need to direct it, it isn't particularly complicated. If you want the longest match, then left factor everything and let it do that:

A ( B (C|) |) ;

And set the token type at the appropriate points.

Jim

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "il-antlr-interest" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [email protected]
For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en
-~----------~----~----~----~------~----~------~--~---





List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: 
http://www.antlr.org/mailman/options/antlr-interest/your-email-address

Reply via email to