Answering my own question :) I wrote a class named ANTLRMemoryMappedFileStream which is nearly a complete ripoff of ANTLRStringStream, except backed by a memory-mapped file. That solved the buffering problem. But now, I get an OutOfMemoryError in the parser with my big test file.
I'm parsing chess games in PGN (portable game notation) format. A game in PGN format is usually under 1k, but a PGN file can contain many games. Most PGN files have several thousand games and those are no problem for Antlr and my grammar. But, PGN files with a million or more games are not rare -- especially in the case where a database user wants to restore an entire collection, or move it to a new chess database management program (like the one I'm working on). So, it's important for me to be able to do parse these huge files. I'm wondering if it's possible for the Parser to notify the Stream that a game has been parsed. At that point, the Stream implementation can flip the buffer. Does that sound like something that might work? Is that possible? On Sun, Oct 4, 2009 at 4:07 AM, Mark Boylan <[email protected]> wrote: > Hi. > > My grammar is working really well with smaller test files, but I run > out of heap space on large files. Unfortunately, my users will expect > to be able to load pretty big files occasionally (~1GB). > > Looking at the code documentation for the Antlr3 stream classes, it > looks like they copy the entire stream. I'm thinking that I need to > write a custom implementation of IntStream or CharStream that buffers > the input. Is that the right way to solve this? Can someone point me > in the right direction? > > Thanks! > > - mark > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---
