On Monday, 4 June 2012 at 21:39:50 UTC, Dmitry Olshansky wrote:
And before you run away with that horrible idea of ever
decoding UTF in lexer... Just don't do that. Trust me, it's not
as small a price as it seems at first. At least keep it only at
prototype stage as it simplifies things.
I didn't plan to convert input into some other encoding. But
missed the idea that it is possible to create finite automata as
a template and avoid decoding altogether. IIRC, I rejected this
approach when decided to convert everything into UTF-8 long ago,
and didn't reconsider after discarding that idea after your
previous suggestion to avoid converting. Thus your idea was used
only partially, and now I wonder how did I not discover this
myself! :)
