Martin Probst wrote:
The idea is that a token is defined as specific type, but can also exist as another type at the same time. So, if the parser needs to know if there is an ID, the code generated asks if the token is a real ID OR has a superimposed ID and so reduces all the complexity of using keywords as identifiers (and probably other case, but this is the specific problem domain), to an extra || operation and an additional LA type operation, something like this: int LA1_0 = input.LA(1); int LA1_0_s = input.LAS(1); if ( (LA1_0==ID)||(LA1_0_s==ID ) { alt1=1; } It still means you need to use a syntactic predicate on occasion, but now that is just to favor the real token over the superimposed token when/if necessary. Say the tokens were defined like this (syntax off the top of my head): ID : 'a'..'z'+ :: KEY1 : 'KEY1'; KEY2 : 'KEY2'; ; When working out what tokens we have, we can see that ID can be a secondary state of some tokens and record this fact (we don't need to know what they are in fact, we just construct those tokens with ID as their secondary type when the lexer runs, and all others have INVALID_TOKEN for secondary type). Now when we generate prediction code, we check to see if the token(s) we are looking for can be a secondary state of any other tokens and generate code accordingly. But if you have: r : KEY2 ( ID | KEY1) .... Then you would only ever get ID. Consequently, predicates would never look at anything but LA and not LAS r: KEY2 ((KEY1)=>KEY1 | ID) ; I think it has to be tried to see what else needs to be covered, such as what happens in LA(*) and so on. Of course, like all good ideas, not only did someone think of it before me, but they even thought it would be funny to call them quantum tokens, even if it isn't particularly accurate from a technical standpoint. All I need is a time machine... Jim --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~--- |
List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
