On Sat, Jul 20, 2024, at 11:51 AM, Tim Düsterhus wrote: > Hi > > On 7/20/24 18:40, Juliette Reinders Folmer wrote: >> Tim, you're making my point for me. This is *exactly* why the current >> change should be reverted. > > I am not sure how you read "PHP users have no idea what a token is" as > an argument in favor of reverting the change, because reverting the > change means that completely reasonable code suddenly stops working with > a parser error in a patch version and PHP users will rightfully come to > PHP's issue tracker to complain. > >> And not, like it is now, an undocumented, random change creating an >> inconsistency in the Tokenizer. > > The tokenizer is doing the right thing: It tokenizes the PHP source > code. It is absolutely normal that PHP first and second-digit updates > make changes to the token stream. New tokens are added, old tokens are > removed, tokens may appear in places where they previously could not > appear for well-formed PHP programs. Tools working on the token stream > need to adapt and this change is no different from any other change to > PHP's syntax in that regard (except that documenting the change was > forgotten). > > Best regards > Tim Düsterhus
Yes, any syntax change means tools need to adapt, but that doesn't mean tokenization can change randomly and accidentally. Syntax changes require an RFC. If an RFC passes that necessitates SA tools update, so be it. (That happens almost every version.) But that's still an RFC change. I would agree with reverting this change for now. The odds of it breaking something for someone are vanishingly small, and it's a bug, not a feature. If we want it to be a feature, we can make an RFC for it. --Larry Garfield
