Actually, I think that if you use UnbufferedTokenStream(), that this will pretty much do what you want already, but it is easy to derive from one of the token streams, and add methods tah can discard buffered tokens once you know you have dealt with them.
Also, if you have comma separated files, then it is usually easier to use awk. Finally, your grammar has myriad lexical ambiguities and I am afraid it is not going to work as you have written it. You cannot have more than one lexer rule that matches the same text as the lexer is not syntax directed, it just tokenizes what it sees. Jim > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Marcin Rzeznicki > Sent: Thursday, April 01, 2010 8:02 AM > To: Kumar, Amitesh > Cc: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > On Thu, Apr 1, 2010 at 4:26 PM, Kumar, Amitesh > <amitesh.ku...@standardbank.com> wrote: > > > > > > > But my general issue is that not all my data is a simple CSV file > some > > will be multi line records. Hence I didn't want to keep a record of > the > > tokens. > > Any ideas . By the way thanks for your reply. > > > > Hi > You can easily implement your own TokenStream that is optimized for > your use case eg. does not try to keep everything in one big array. If > you explore this possibility, you will quickly discover that it is > very easy thing to do and test. Hope it helps. > > > > Cheers > > Kumaap0 > > > -- > Greetings > Marcin Rzeźnicki > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
