Terence, Can you point me a some more details about "traditional lex-like system including lexical modes in v4"
I'm don't know enough about scannerless parsering, are there details on how it support island grammars? Thanks for the response. Regards Gary > ... traditional lex-like system including lexical modes in v4. > > To deal with the island grammars and context-sensitive Lexing issues, I'm > supporting scannerless parsers in v4. > > The reason I wrote the StringTemplate lexer by hand was partially due to the > island grammar issue, but primarily because the delimiters can be dynamically > specified by the user. That tends to defeat any automata-based systems for > lexing. > > Ter > > On Aug 30, 2011, at 7:40 PM, Gary Miller wrote: > >> Hey All, >> >> Slightly off topic post, I thought there might be some interest. >> >> Last night I went to a talk by Rob Pike of Google, you can watch the talk at >> https://www.youtube.com/watch?v=HxaD_trXwRE&feature=player_embedded >> >> Before I went my thinking was that this could probably be knocked up >> in ANTLR in a few minutes, but then .... >> All the uncomfort I have with ANTLR lexering came back to me. >> So I though I'd go to the source and have a look at the lexer for ST, >> and low and behold ST's lexer is written by hand. >> Now I'm feeling quite uncomfortable about ANTLR's lexing. >> >> I think it basically comes down to the stateless nature of the ANTLR lexing. >> Not the first time context-sensitive scanning has been mentioned on >> the list (*). >> Yes I know that it can be made statefull (*) and/or I can push more >> onto the parser, but both of these have issues. >> Statefull ANTLR lexing code I generally find more confusing and harder >> to write then functionally equivalent code in a target language. >> Pushing more into the parser in this particular case is inefficient as >> there are large chunks of text that doesn't need to be tokenized and >> there is the issue the whitespace tokens might need to behaving >> differently in different places (hidden verse not). >> >> >> Started off as an off topic post, ended as a rant about lexing regards >> Gary >> P.S. I've started on the ANTLR target for Go, still very immature. >> https://github.com/millergarym/antlr/tree/, >> >> * Scott Stanchfield's context-sensitive scanning >> http://javadude.com/articles/antlr-context-sensitive-scanner.html >> >> * a good example of this is Jim's numerical lexing for JavaFX >> http://www.antlr.org/wiki/display/ANTLR3/Lexer+grammar+for+floating+point,+dot,+range,+time+specs >> >> List: http://www.antlr.org/mailman/listinfo/antlr-interest >> Unsubscribe: >> http://www.antlr.org/mailman/options/antlr-interest/your-email-address > > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
