Just replace new CommonTokenStream(...). But you will need to fix your grammar before it will all work of course.
Jim > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Kumar, Amitesh > Sent: Thursday, April 01, 2010 8:52 AM > To: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > Great where is this UnbufferedTokenStream. > > Cheers Jim > > > Amitesh Kumar |CIB Integration | Business Infrastructure Technology | > Standard Bank CIB International | Ground Floor, 20 Gresham Street, > London, EC2V 7JE > T: +44 [0]203 145 5575 | E: amitesh.ku...@standardbank.com > > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Jim Idle > Sent: 01 April 2010 16:27 > Cc: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > Actually, I think that if you use UnbufferedTokenStream(), that this > will pretty much do what you want already, but it is easy to derive > from one of the token streams, and add methods tah can discard buffered > tokens once you know you have dealt with them. > > > Also, if you have comma separated files, then it is usually easier to > use awk. Finally, your grammar has myriad lexical ambiguities and I am > afraid it is not going to work as you have written it. You cannot have > more than one lexer rule that matches the same text as the lexer is not > syntax directed, it just tokenizes what it sees. > > Jim > > > -----Original Message----- > > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > > boun...@antlr.org] On Behalf Of Marcin Rzeznicki > > Sent: Thursday, April 01, 2010 8:02 AM > > To: Kumar, Amitesh > > Cc: antlr-interest@antlr.org > > Subject: Re: [antlr-interest] Parsing Large Files > > > > On Thu, Apr 1, 2010 at 4:26 PM, Kumar, Amitesh > > <amitesh.ku...@standardbank.com> wrote: > > > > > > > > > > > But my general issue is that not all my data is a simple CSV file > > some > > > will be multi line records. Hence I didn't want to keep a record of > > the > > > tokens. > > > Any ideas . By the way thanks for your reply. > > > > > > > Hi > > You can easily implement your own TokenStream that is optimized for > > your use case eg. does not try to keep everything in one big array. > If > > you explore this possibility, you will quickly discover that it is > > very easy thing to do and test. Hope it helps. > > > > > > > Cheers > > > Kumaap0 > > > > > > -- > > Greetings > > Marcin Rzeźnicki > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > > Unsubscribe: http://www.antlr.org/mailman/options/antlr- > interest/your- > > email-address > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address > > > *********************************************************************** > ****** > More information on Standard Bank is available at www.standardbank.com > > Everything in this email and any attachments relating to the official > business > of Standard Bank Group Limited and any or all subsidiaries, ("the > Company"), is > proprietary to the Company. It is confidential, legally privileged and > protected > by relevant laws. The Company does not own and endorse any other > content. > Views and opinions are those of the sender unless clearly stated as > being > that of the Company. > > The person or persons addressed in this email are the sole authorised > recipient. Please notify the sender immediately if it has > unintentionally, > or inadvertently reached you and do not read, disclose or use the > content > in any way and delete this e-mail from your system. > > The Company cannot ensure that the integrity of this email has been > maintained nor that it is free of errors, virus, interception or > interference. > The sender therefore does not accept liability for any errors or > omissions > in the contents of this message which arise as a result of e-mail > transmission. > If verification is required please request a hard-copy version. This > message > is provided for informational purposes and should not be construed as a > solicitation or offer to buy or sell any securities or related > financial instruments. > *********************************************************************** > ****** > > > This message has been scanned for viruses by BlackSpider MailControl - > www.blackspider.com > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
