David Beazley wrote:
lexer should be a generator that emits a stream of tokens. It should then be
possible to
process/filter that stream with other generators/iterators as needed (for
example, using various
functions in itertools). The only reason PLY doesn't use generators is that
they didn't exist when
it was first written (2001).
*THE* thing I'd like to see happening, is to have streaming support of the lexer, that is, I can
give it a file-handle to a 50GB text file, and it won't try loading that file data in memory.
Unfortunately, that would probably mean rewriting RE, which is unlikely to
happen :(
The only other big feature is that I might get around to finishing is the
ply/cpp.py module
(currently about 95% implemented). What is the cpp.py module you ask? Well,
it's eventually going to
be a fully working C preprocessor--complete with support for macros and
everything else. I'd like to
finish it simply because having a pure Python C preprocessor would be cool.
I didn't know that existed. I might be interested in that.
Do you have an URL? what features are you missing?
In a processing program I am helping to write, the input is currently being pre-processed by 'gcc
-E' first (you may have seen the Position object troubles with lineno that I had in the bug tracker).
Using your code may give a cleaner solution, if you can use ply/cpp.py as a pre-processing step
before the normal lexer.
Albert
--
You received this message because you are subscribed to the Google Groups
"ply-hack" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/ply-hack?hl=en.
