On 09/12/2013 03:30 AM, deadalnix wrote:
That's correct, but that implies re-lexing the tokens, which has
negative performance implications.
Indeed. What solution do you have in mind ?
Buffering the tokens would work. There are some ways promote input
ranges to forward ranges. But there are also some pitfalls like the
implicit save on copy.
I have two prototypes for a generic input range buffer.
https://gist.github.com/dawgfoto/2187220 - uses growing ring buffer
https://gist.github.com/dawgfoto/1257196 - uses ref counted lookahead
buffers in a singly linked list
The lexer itself has a ringbuffer for input ranges.
https://github.com/Hackerpilot/phobos/blob/master/std/d/lexer.d#L2278
