On 20.07.2024 at 18:51, Tim Düsterhus wrote: > >> And not, like it is now, an undocumented, random change creating an >> inconsistency in the Tokenizer. > > The tokenizer is doing the right thing: It tokenizes the PHP source > code. It is absolutely normal that PHP first and second-digit updates > make changes to the token stream. New tokens are added, old tokens are > removed, tokens may appear in places where they previously could not > appear for well-formed PHP programs. Tools working on the token stream > need to adapt and this change is no different from any other change to > PHP's syntax in that regard (except that documenting the change was > forgotten).
If the tokenizer would tokenize a whole file as a single token, would that also be correct? Of course, I'm exaggerating, but <https://3v4l.org/qIf2c> doesn't look correct to me – "yield /* comment */ from" shouldn't be a single token. Cheers, Christoph
