On Thu, Feb 24, 2011 at 3:25 AM, Richard Druce <[email protected]> wrote: > I have a question on the general best practice and speed between of > using tokens vs rules to construct parts of the grammar. Our language > has many phrases that share words, a simplified sample being 'first' > and 'first second'. Would I be better off putting them in as tokens > or rules from a speed of parsing perspective. Some of my tokens also > contain whitespace. > > i.e > rule1: FIRST; > > rule2: FIRST WS SECOND; > > FIRST: 'first'; > SECOND: 'second' > > WS: ' '; > > or > > rule1: RULE1; > > rule2: RULE2; > > RULE1: 'first second'; > > RULE2: 'first'; > > Thanks, > > Richard
I'm no expert on language design, but this sounds like way premature optimization to me. Why not focus on what the language purpose is, ensure that the grammar is effective and efficient and capturing it, and then revisit it about speed of tokenization? In general, ANTLR is pretty darn speedy, and if you are doing much complex with the language the tokenization is unlikely to be the slow part. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
