On 21-7-2024 17:11, Rowan Tommins [IMSoP] wrote:
Oddly, the crux of this debate seems to be less that all options have
major impact, and more that none of them do. If the implementation had
caused a serious security or performance regression, I don't think
we'd have any hesitation in backing it out if a better implementation
wasn't ready; or contrarily, if it had been a headline new feature, we
wouldn't even be considering that option.
The crux - to me - is that it is an undocumented breaking change, which
by definition is a bug.
As I've said before, I'm not against changing the tokenization, what I'm
speaking up about is that it was done in an inconsistent, semi-random
and undocumented way.
As it is, an improved implementation *is* proposed:
https://github.com/php/php-src/pull/15041 Comparing this purely
against 8.2, it seems a reasonable compromise: consumers of the token
stream still need to make a change, but it's the slightly more
intuitive one of "treat yield and from as separate tokens, which might
have other tokens between". Personally, I think it's reasonable to
consider that a bug fix to the original change, and push it into 8.3.
Projects using the token stream could detect the versions with the
imperfect implementation, and emit an error explaining the "bug" if a
T_YIELD_FROM token doesn't match /yield\s+from/ Projects parsing the
source code on their own will have to handle this however they handle
contexts where comments have always been allowed.
I agree the improved tokenization proposed in the PR makes better sense.
Having said that, it will make the breaking change much much larger as
it now would no longer just be a change which can be handled in the
PHPCS token stream, but one which will impact individual sniffs, what
with the removal and introduction of new tokens.
If that PR gets merged, it will mean that PHPCS will need to undo the
PHP 8.3 and PHP 8.4 tokenization for the time being (in the 3.x major)
and will only "polyfill" the new tokenization as of PHPCS 4.0. In
practice, we would be moving the breaking change to PHPCS 4.0 version to
not break sniffs.
Smile,
Juliette
