Then yes, I suggest that you implement a custom input stream that allows you to push input in to a FIFO, which is always read before the supplied input stream. Of course you will need a stack of these because of nested macros and so on. You would need to override LA and related methods but it should not be a big deal.
Jim > -----Original Message----- > From: [email protected] [mailto:antlr-interest- > [email protected]] On Behalf Of Alex Marin > Sent: Wednesday, August 18, 2010 10:39 AM > To: [email protected] > Cc: [email protected] > Subject: Re: [antlr-interest] ANTLR 2.7.7 - prepend to InputBuffer > > Hi Jim, > > You are right, this would be the straightforward way to handle the problem in > its entirety. However, we wouldn't want to rewrite the preprocessing part of > the lexer, as it works fine except for the cases I've mentioned. Right now we > hope that implementing an InputBuffer which accepts its input to be > modified during lexing would be of less impact to what we already have. > > As for the language users - in my opinion they will always _use_ the language > in _all_ possible ways, no matter how unreadable or twisted this would > render the code. If anyone could ever change this, I think it's the language > designers, not the parser implementers :) > > Thanks for your suggestion anyway! > Alex > > On 08/18/2010 06:29 PM, Jim Idle wrote: > > The better solution is to create a separate pre-processor stage. You > > run the pre-processor and it feed its output in to the 'real' lexer. > > This is a lot easier to maintain and by the time you had messed around > > trying to create secondary lexers, include stacks and so on for other > > solutions, you end up with this being faster. Also, the ability to > > have a much simpler lexer that is just handling #if, #define and macro > > expansion makes things much easier (other than macros in macros). > > > > Now, if your environment is known,then you could just implement this > > in m4, which is available just about everywhere. > > > > http://en.wikipedia.org/wiki/Preprocessor > > http://en.wikipedia.org/wiki/M4_(computer_language) > > http://www.gnu.org/software/m4/ > > > > You could also just run the C pre-processor and define your language > > as using the C macro engine. > > > > And further, if you can do without macros altogether, you can stop the > > language users creating unreadable, undebuggable code because they > > thought macros were cool and clever and used them nested to 23 levels > > ;-) > > > > Jim > > > > > > > > > >> -----Original Message----- > >> From: [email protected] [mailto:antlr-interest- > >> [email protected]] On Behalf Of Alex Marin > >> Sent: Wednesday, August 18, 2010 8:00 AM > >> To: [email protected] > >> Cc: [email protected] > >> Subject: [antlr-interest] ANTLR 2.7.7 - prepend to InputBuffer > >> > >> Hello everyone, > >> > >> Is it possible in ANTLR 2.7.7 to add characters on the fly to a > >> lexer's InputBuffer? Do you have any suggestion/guideline/experience > >> on how to do this? (without re-implementing the InputBuffer/CharQueue > >> classes) > >> > >> I will explain the reason behind the need to do this. The language we > >> are lexing allows pre-processing, by using C-like defines (or macros). > >> > >> // Define foo macro > >> `define foo this is the replacement of foo > >> > >> // Use foo macro > >> `foo // this expands to 'this is the replacement of foo' > >> > >> // Define moo macro with a parameter > >> `define moo(x) x+2 > >> > >> // Use moo macro > >> `moo(1) // this expands to '1+2' > >> > >> At the moment, when our lexer encounters a macro, it starts another > >> lexer which lexes the replacement. E.g: > >> > >> ... > >> `define foo this is the replacement of foo ... > >> `foo // Here another lexer is started and provided the input stream > >> 'this > >> > > is > > > >> the replacement of foo' > >> ... > >> > >> This approach works very well at the time being, except for the > >> situations when the replacement text is not a fully "lexable" piece > >> of code, like in > >> > > the > > > >> example below: > >> > >> ... > >> `define a(x,y) x+y // Define macro a with parameters x and y `define > >> b > >> > > `a(1 // > > > >> Define macro b without parameters ... > >> `b,2) // Use macro b; this should expand to `a(1,2) and therefore to > >> 1+2 > >> > > ... > > > >> Now, the problem is that the lexer we start for `b will lex '`a(`' > >> which > >> > > is not > > > >> lexically correct and will fail with an error. > >> > >> The best solution would be to be able to insert in the main lexer's > >> buffer > >> > > the > > > >> replacement of `b and continue lexing normally. Of course, no new > >> lexer would be needed then. So, after matching `b, the buffer would > >> look like > >> `a(1,2) > >> > >> Thanks in advance, > >> Alex > >> > >> > >> List: http://www.antlr.org/mailman/listinfo/antlr-interest > >> Unsubscribe: > >> http://www.antlr.org/mailman/options/antlr-interest/your- > >> email-address > >> > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > > Unsubscribe: > > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
