On 20-7-2024 18:04, Tim Düsterhus wrote:
PHP users have no idea what a token is internally. I'm looking at this
from a PHP user perspective. It looks like two keywords, it walks like
two keywords and it quacks like two keywords. I find it reasonable for
users to consider this as two keywords and not care about how it's
implemented internally.
So, what "feels" natural (whitespace-wise) to one person may not be the
same for the next, but comments _within_ tokens is different thing and
should in my opinion, not be allowed.
As I've said: I agree that the current situation is unfortunate. But
the correct solution is not "disallow comments", but "split the
T_YIELD_FROM into T_YIELD T_WHITESPACE T_FROM_FOR_YIELD_FROM".
Tim, you're making my point for me. This is *exactly* why the current
change should be reverted.
I'm not against changing the tokenization of "yield from" and the GH
ticket thread also contains an alternative proposal for this from Bob
[1], but like Matthew also said [2]: if that's something we want to do,
let's have a proper discussion about it and let it go through an RFC and
be a documented change.
And not, like it is now, an undocumented, random change creating an
inconsistency in the Tokenizer.
Smile,
Juliette
1: https://github.com/php/php-src/issues/14926#issuecomment-2228855422
2: https://externals.io/message/124462#124515
