On 02/28/2013 05:48 PM, Dmitry Olshansky wrote:
...line 300: case 0x1A: // ^Z means end of file case 0: break; On the lines you noted it claimed that that 0x1a is outdate. Along with the fact that you allocate filesize+2 and fill the last 2 bytes with zeros. In any case I see 0 and 0x1a as 2 values that act like sentinels i.e. a tuple. And this is what spec says - any one of them is a sentinel. Correct me if I'm wrong.
A sentinel is some data the original data is augmented with in order to simplify its processing.
The lexer acts the same on 0x1A and 0, but only the additional 0 at the end which does not occur in the input is the sentinel. The lexer may even encounter a 0 that is not a sentinel.
