On 8/17/07, Tor Norbye <[EMAIL PROTECTED]> wrote: > On Aug 15, 2007, at 1:15 PM, Thomas E Enebo wrote: > > > I implemented a patch which allows a bytelist to be read directly by a > > lexersource. This ends up with the following (plusses and minuses): > > > > + We do not convert the bytelist to a String in order to parse > > during evals > > + We do not save a copy of that converted String in ByteList (bytelist > > caching mechanism) > > + I discovered an issue with our caching logic which is a little scary > > (e.g. a bug) > > - Evals are the same speed or a little slower because of the > > abstraction of lexing > > > > One thing which sucks about the patch is that the ByteListLexerSource > > could not just be a bytelist and an index. It needs to defer back to > > a bunch of special logic in the base class. To make a long > > explanation short, the lexer sometimes pushes back data to the source. > > What it pushes back isn't guaranteed to be identical to original > > source (think \r\n and \n). With quite a bit more work I could maybe > > work around this, but the newline handling stuff sucks. > > It's also done for heredocs, where the lexer will process the input > out of order (everything following the first heredoc marker on the > line is pushed into the stream -after- the actual string content) so > just special casing newlines won't work.
Actually heredocs was the example I was thinking of when I wrote that. Though I believe it can happen elsewhere too. -Tom -- Blog: http://www.bloglines.com/blog/ThomasEEnebo Email: [EMAIL PROTECTED] , [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email
