Ah - this is still in development. In which case you would need to extend CommonTokenStream and create your own system. Or you could switch the development snapshot.
Jim > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Kumar, Amitesh > Sent: Thursday, April 01, 2010 9:07 AM > To: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > Ive got ANTLR version 3.2 it doesn't seem to have > UnbufferedTokenStream, there doesn't seem to be a newer version on the > site > > Amitesh Kumar |CIB Integration | Business Infrastructure Technology | > Standard Bank CIB International | Ground Floor, 20 Gresham Street, > London, EC2V 7JE > T: +44 [0]203 145 5575 | E: amitesh.ku...@standardbank.com > > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Jim Idle > Sent: 01 April 2010 16:55 > To: antlr-interest@antlr.org > Subject: Re: [antlr-interest] Parsing Large Files > > Just replace new CommonTokenStream(...). > > But you will need to fix your grammar before it will all work of > course. > > Jim > > > -----Original Message----- > > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > > boun...@antlr.org] On Behalf Of Kumar, Amitesh > > Sent: Thursday, April 01, 2010 8:52 AM > > To: antlr-interest@antlr.org > > Subject: Re: [antlr-interest] Parsing Large Files > > > > Great where is this UnbufferedTokenStream. > > > > Cheers Jim > > > > > > Amitesh Kumar |CIB Integration | Business Infrastructure Technology | > > Standard Bank CIB International | Ground Floor, 20 Gresham Street, > > London, EC2V 7JE > > T: +44 [0]203 145 5575 | E: amitesh.ku...@standardbank.com > > > > -----Original Message----- > > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > > boun...@antlr.org] On Behalf Of Jim Idle > > Sent: 01 April 2010 16:27 > > Cc: antlr-interest@antlr.org > > Subject: Re: [antlr-interest] Parsing Large Files > > > > Actually, I think that if you use UnbufferedTokenStream(), that this > > will pretty much do what you want already, but it is easy to derive > > from one of the token streams, and add methods tah can discard > > buffered tokens once you know you have dealt with them. > > > > > > Also, if you have comma separated files, then it is usually easier to > > use awk. Finally, your grammar has myriad lexical ambiguities and I > am > > afraid it is not going to work as you have written it. You cannot > have > > more than one lexer rule that matches the same text as the lexer is > > not syntax directed, it just tokenizes what it sees. > > > > Jim > > > > > -----Original Message----- > > > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > > > boun...@antlr.org] On Behalf Of Marcin Rzeznicki > > > Sent: Thursday, April 01, 2010 8:02 AM > > > To: Kumar, Amitesh > > > Cc: antlr-interest@antlr.org > > > Subject: Re: [antlr-interest] Parsing Large Files > > > > > > On Thu, Apr 1, 2010 at 4:26 PM, Kumar, Amitesh > > > <amitesh.ku...@standardbank.com> wrote: > > > > > > > > > > > > > > > But my general issue is that not all my data is a simple CSV file > > > some > > > > will be multi line records. Hence I didn't want to keep a record > > > > of > > > the > > > > tokens. > > > > Any ideas . By the way thanks for your reply. > > > > > > > > > > Hi > > > You can easily implement your own TokenStream that is optimized for > > > your use case eg. does not try to keep everything in one big array. > > If > > > you explore this possibility, you will quickly discover that it is > > > very easy thing to do and test. Hope it helps. > > > > > > > > > > Cheers > > > > Kumaap0 > > > > > > > > > -- > > > Greetings > > > Marcin Rzeźnicki > > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > > > Unsubscribe: http://www.antlr.org/mailman/options/antlr- > > interest/your- > > > email-address > > > > > > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > > Unsubscribe: http://www.antlr.org/mailman/options/antlr- > interest/your- > > email-address > > > > > > > ********************************************************************** > > * > > ****** > > More information on Standard Bank is available at > www.standardbank.com > > > > Everything in this email and any attachments relating to the official > > business of Standard Bank Group Limited and any or all subsidiaries, > > ("the Company"), is proprietary to the Company. It is confidential, > > legally privileged and protected by relevant laws. The Company does > > not own and endorse any other content. > > Views and opinions are those of the sender unless clearly stated as > > being that of the Company. > > > > The person or persons addressed in this email are the sole authorised > > recipient. Please notify the sender immediately if it has > > unintentionally, or inadvertently reached you and do not read, > > disclose or use the content in any way and delete this e-mail from > > your system. > > > > The Company cannot ensure that the integrity of this email has been > > maintained nor that it is free of errors, virus, interception or > > interference. > > The sender therefore does not accept liability for any errors or > > omissions in the contents of this message which arise as a result of > > e-mail transmission. > > If verification is required please request a hard-copy version. This > > message is provided for informational purposes and should not be > > construed as a solicitation or offer to buy or sell any securities or > > related financial instruments. > > > ********************************************************************** > > * > > ****** > > > > > > This message has been scanned for viruses by BlackSpider MailControl > - > > www.blackspider.com > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > > Unsubscribe: http://www.antlr.org/mailman/options/antlr- > interest/your- > > email-address > > > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
