On Aug 15, 2007, at 1:15 PM, Thomas E Enebo wrote:
I implemented a patch which allows a bytelist to be read directly by a
lexersource. This ends up with the following (plusses and minuses):
+ We do not convert the bytelist to a String in order to parse
during evals
+ We do not save a copy of that converted String in ByteList (bytelist
caching mechanism)
+ I discovered an issue with our caching logic which is a little scary
(e.g. a bug)
- Evals are the same speed or a little slower because of the
abstraction of lexing
One thing which sucks about the patch is that the ByteListLexerSource
could not just be a bytelist and an index. It needs to defer back to
a bunch of special logic in the base class. To make a long
explanation short, the lexer sometimes pushes back data to the source.
What it pushes back isn't guaranteed to be identical to original
source (think \r\n and \n). With quite a bit more work I could maybe
work around this, but the newline handling stuff sucks.
It's also done for heredocs, where the lexer will process the input
out of order (everything following the first heredoc marker on the
line is pushed into the stream -after- the actual string content) so
just special casing newlines won't work.
-- Tor
---------------------------------------------------------------------
To unsubscribe from this list please visit:
http://xircles.codehaus.org/manage_email
