On 10/02/2013 04:41 PM, Dicebot wrote:
After brief discussion with Brian and gathering data from the review
thread, I have decided to start voting for `std.d.lexer` inclusion into
Phobos.
I also have to vote with no for now.
My biggest concern is that the lexer incorporates a string pool,
something that isn't strictly part of lexing.
IMO this is a major design flaw and possible performance/memory issue.
It is buried into the API because byToken takes const(byte)[], i.e.
mutable data, but each Token carries a string value, so it always
requires a copy.
For stream oriented lexing, e.g. token highlighting, no string pool is
required at all.
Instead the value type of Token should be something like
take(input.save, lengthOfToken).
Why was the Tok!">>=", Tok!"default" idea turned down. This leaves us
with undesirable names like Tok.shiftRightAssign, Tok.default_.
There are a few smaller issues that haven't yet been addressed, but of
course this can be done during the merge code review.
Adding it as experimental module would be a good idea.
