On Tuesday, 8 October 2013 at 00:16:45 UTC, Andrei Alexandrescu
wrote:
To put my money where my mouth is, I have a proof-of-concept
tokenizer for C++ in working state.
http://dpaste.dzfl.pl/d07dd46d
Why do you use "\0" as end-of-stream token:
/**
* All token types include regular and reservedTokens, plus the
null
* token ("") and the end-of-stream token ("\0").
*/
We can have situation when the "\0" is a valid token, for example
for binary formats. Is it possible to indicate end-of-stream
another way, maybe via "empty" property for range-based API?
