Fabien, Look at the generated code and the 'constructor' for your lexer. If you trace through that code you will see that the mTokens() is just a static function, the pointer to which is installed in the lexer structures. You ca install your own.
However, pre-processors are easy enough and though you have to add extra code to your lexer rules, it is much easier (and more maintainable) to set the channel of the token to a member variable value or call SKIP. Then implement the pre-processor in the lexer. Sometimes you cannot do this though. For instance the C# pre-processor is implemented in the lexer, but the VB pre-processor is its own up front program (which I implemented as a parser grammar myself). In your case though, I don't think that you want to override mToken(), you want to override the nextToken and nextTokenStr functions. Just copy them from antlr3lexer.c, modify as needed and perform the SKIP operation when your flag is set to 'off'. You then install the pointer to your version of nextToken() in the lexer structures after you have created your lexer and before anything calls in to it. As I say though, personally I think it is neater to explicitly code for this in the lexer rules. Jim -----Original Message----- From: Fabien Antoine [mailto:[email protected]] Sent: Wednesday, March 03, 2010 12:02 AM To: Jim Idle Subject: Re: [antlr-interest] Extends the lexer in C target Hi Jim and thanks for your answer. I saw all what you said concerning the lexer implementation and it's precisely the problem. I want to create a kind of preprocessor that interprets statements only if a constant is defined. And I don't want to manage a Preprocessor token in all my rules. I tried to reproduce the system used in the C# parser done by Andrew Bradnan but its output target is C#. Then he can extend the generated lexer and override the "mTokens" function. By overriding the "mTokens" function, he can push the next token into the hidden channel if the preprocessor decided that current statements must not be returned. Then the only thing I want to do is to add this "filter" to the lexer. I'm fully open to other "cleaner" solutions... Fabien On 02/03/2010 16:53, Jim Idle wrote: > mTokens is generated but you can install any function you like in its place > by adding code in the initialization. You should see that the address of the > static function is just stored in the lexer structures. Just replace the > pointer with a pointer to your own function. Read through the source though > to find out what you need to do in that function and the functions that it > invokes. > > That said, if you are trying to override this function, then you are really > replacing the lexer. Perhaps you should tell us why you want to override that > and what you are trying to achieve - maybe there is a better way. > > Jim > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
