Jim, what about the imaginary tokens I create in the parser? From what you write it seems I have to include them in my hand-crafted .tokens files, but which .token file?
Thanks, Luigi On 15 Apr 2010, at 21:51, Jim Idle wrote: > Your best bet is to pre-create the .tokens file and use the tokenVocab option > in the lexer. Just make sure that you keep token numbers contiguous and don't > leave gaps. The token numbers will then not only be shared, but known values. > You can have more tokens than you actually generate or deal with in a > particular parser. I have used this concept across SQL parsers for instance > so that a target enging can be origin agnostic. > > Without a vocab option, the lexer will parse and produce the .tokens file. > This will be imported by the parsers, which will produce their own .tokens > files, which can be imported by tree walkers and so on but when they add > tokens (even if they are the same names), then they will not be shared by > different parsers. > > You could probably get the outcome you want by carefully controlling the > build order and setting the tokenVocab and so on, but it is trivial to > maintain your own. > > Jim > >> -----Original Message----- >> From: [email protected] [mailto:antlr-interest- >> [email protected]] On Behalf Of Luigi Iannone >> Sent: Thursday, April 15, 2010 6:45 AM >> To: ANTLR >> Subject: [antlr-interest] Imported parser tokens mismatch >> >> Hi, >> >> I am struggling to re-use tokens across grammars. I have two combined >> grammars, namely >> >> ManchesterOWLSyntax.g >> OPPLScript.g >> >> They both import a parser grammar called MOWLParser.g, >> >> ManchesterOWLSyntax.g imports it directly, whereas OPPLScript.g >> imports OPPLParser.g which imports MOWLParser.g >> >> I would have thought that all the .tokens generated files would overlap >> on all the tokens defined inside MOWLParser.g >> >> Maybe I am missing something but if I try to generate the code I end up >> with different MOWLParser.tokens which results in different token type >> codes for the same token. >> >> Just as example the token CONJUNCTION gets code 77 when generating >> OPPLScript.g and 56 when generating ManchesterSyntax.g >> >> I put all the relevant files at >> http://www.cs.man.ac.uk/~iannonel/antlrexperiments/ (both the loose >> files and their zipped archive: import.zip) >> >> Thanks in advance for any help, >> >> Luigi >> >> List: http://www.antlr.org/mailman/listinfo/antlr-interest >> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- >> email-address > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
