Great where is this UnbufferedTokenStream. Cheers Jim
Amitesh Kumar |CIB Integration | Business Infrastructure Technology | Standard Bank CIB International | Ground Floor, 20 Gresham Street, London, EC2V 7JE T: +44 [0]203 145 5575 | E: amitesh.ku...@standardbank.com -----Original Message----- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Jim Idle Sent: 01 April 2010 16:27 Cc: antlr-interest@antlr.org Subject: Re: [antlr-interest] Parsing Large Files Actually, I think that if you use UnbufferedTokenStream(), that this will pretty much do what you want already, but it is easy to derive from one of the token streams, and add methods tah can discard buffered tokens once you know you have dealt with them. Also, if you have comma separated files, then it is usually easier to use awk. Finally, your grammar has myriad lexical ambiguities and I am afraid it is not going to work as you have written it. You cannot have more than one lexer rule that matches the same text as the lexer is not syntax directed, it just tokenizes what it sees. Jim > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Marcin Rzeznicki > Sent: Thursday, April 01, 2010 8:02 AM > To: Kumar, Amitesh > Cc: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > On Thu, Apr 1, 2010 at 4:26 PM, Kumar, Amitesh > <amitesh.ku...@standardbank.com> wrote: > > > > > > > But my general issue is that not all my data is a simple CSV file > some > > will be multi line records. Hence I didn't want to keep a record of > the > > tokens. > > Any ideas . By the way thanks for your reply. > > > > Hi > You can easily implement your own TokenStream that is optimized for > your use case eg. does not try to keep everything in one big array. If > you explore this possibility, you will quickly discover that it is > very easy thing to do and test. Hope it helps. > > > > Cheers > > Kumaap0 > > > -- > Greetings > Marcin Rzeźnicki > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address ***************************************************************************** More information on Standard Bank is available at www.standardbank.com Everything in this email and any attachments relating to the official business of Standard Bank Group Limited and any or all subsidiaries, ("the Company"), is proprietary to the Company. It is confidential, legally privileged and protected by relevant laws. The Company does not own and endorse any other content. Views and opinions are those of the sender unless clearly stated as being that of the Company. The person or persons addressed in this email are the sole authorised recipient. Please notify the sender immediately if it has unintentionally, or inadvertently reached you and do not read, disclose or use the content in any way and delete this e-mail from your system. The Company cannot ensure that the integrity of this email has been maintained nor that it is free of errors, virus, interception or interference. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. ***************************************************************************** This message has been scanned for viruses by BlackSpider MailControl - www.blackspider.com List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
