[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Brecht Machiels added the comment: In order to adapt code to this change, can we assume that a NEWLINE token with an empty string only occurs right before the ENDMARKER? -- nosy: +brechtm ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Aaron Meurer added the comment: Is it expected behavior that comments produce NEWLINE if they don't have a newline and don't produce NEWLINE if they do (that is, '# comment' produces NEWLINE but '# comment\n' does not)? -- nosy: +asmeurer ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Gregory P. Smith added the comment: https://bugs.python.org/issue35107 filed to track further fallout from this API change. -- ___ Python tracker ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tal Einat : -- versions: -Python 2.7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Terry J. Reedy : -- pull_requests: -9424 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tim Graham : -- pull_requests: +9424 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ned Deily : -- pull_requests: -9411 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ned Deily : -- pull_requests: -9415 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tim Graham : -- pull_requests: +9415 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Benjamin Peterson added the comment: New changeset a1f45ec73f0486b187633e7ebc0a4f559d29d7d9 by Benjamin Peterson (Tal Einat) in branch '2.7': bpo-33899: Revert tokenize module adding an implicit final NEWLINE (GH-10072)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Terry J. Reedy added the comment: #33766 was about documenting the C tokenizer change, some years ago, that made end-of-file EOF and end-of-string EOS generate the NEWLINE token required to properly terminate statements. "The end of input also serves as an implicit terminator for the final

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Dong-hee Na : -- pull_requests: +9411 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: Thanks for helping with the fallout from this, Gregory. -- ___ Python tracker ___ ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

miss-islington added the comment: New changeset b4c9874f5c7f64e1d41cbc588e515b8851bbb90c by Miss Islington (bot) (Tal Einat) in branch '3.7': [3.7] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) (GH-10074)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

miss-islington added the comment: New changeset 9a0476283393f9988d0946491052d7724a7f9d21 by Miss Islington (bot) (Tal Einat) in branch '3.6': [3.6] bpo-33899: Mention tokenize behavior change in What's New (GH-10073) (GH-10075)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tal Einat : -- pull_requests: +9409 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tal Einat : -- pull_requests: +9408 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Gregory P. Smith added the comment: New changeset dfba1f67e7f1381ceb7cec8fbcfa37337620a9b0 by Gregory P. Smith (Tal Einat) in branch 'master': bpo-33899: Mention tokenize behavior change in What's New (GH-10073)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Gregory P. Smith added the comment: some pylint fallout appears to be addressed in https://github.com/PyCQA/pylint/commit/2698cbe56b44df7974de1c3374db8700296c6fad -- ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: See PR GH-10073 adding mention in "What's New". -- ___ Python tracker ___ ___ Python-bugs-list

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tal Einat : -- pull_requests: +9407 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Gregory P. Smith added the comment: FYI, An example of other fallout from this change - patsy broke and needed this fix: https://github.com/pydata/patsy/commit/4f53bbaf58c0bf1a9bed73fc67c7c6d0aa7f4e20#diff-53c70e68c6dfd4fe9b08427792cb2bd6 -- nosy: +gregory.p.smith

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: See PR GH-10072 for reverting in 2.7. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Tal Einat : -- pull_requests: +9406 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Benjamin Peterson added the comment: Please revert in 2.7. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Ned Deily added the comment: I don't have a strong opinion about 2.7 here. Ultimately, it's Benjamin's call. But it might make sense to revert for 2.7 since it hasn't been released yet. -- nosy: +benjamin.peterson ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: Ned, should this also be added to the 2.7 What's New? Or perhaps reverted on the 2.7 branch? -- ___ Python tracker ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: I'm sorry to have caused this mess, it was bad judgement on my part. Adding mention in What's is a good idea, Ned, I'll do that. -- ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Ned Deily added the comment: Apparently this change also affected IPython. Perhaps we should add an entry to the whatsnew documents for 3.7.1 and 3.7.6: https://docs.python.org/3/whatsnew/3.7.html#notable-changes-in-python-3-7-1

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Anthony Sottile added the comment: I'm surprised this was classified as a bug! Though that's subjective so I get that it's difficult to decide what is and what isn't ¯\(ツ)/¯ -- ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: This was backported since it was considered a bug, but you are right that it broke backwards compatibility, and perhaps shouldn't have been backported. Still, with 3.6.6 and 3.7.1 now released, that ship has sailed. We could perhaps revert this on the 2.7

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Anthony Sottile added the comment: This change in behaviour is breaking pycodestyle: https://github.com/PyCQA/pycodestyle/issues/786 Perhaps it shouldn't have been backported (especially all the way to python2.7?) -- nosy: +Anthony Sottile ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: Thanks for all of your work on this, Ammar! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: +Python 2.7, Python 3.6, Python 3.7 ___ Python tracker

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: New changeset 7829bba45d0e2446f3a0ca240bfe46959f01071e by Tal Einat (Ammar Askar) in branch '2.7': [2.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (#8133)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: New changeset 11c36a3e16f7fd4e937466014e8393ede4b61a25 by Tal Einat (Ammar Askar) in branch '3.6': [3.6] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8134)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: New changeset ab75d9e4244ee24bc96ea9d52362899e3bf365a2 by Tal Einat (Ammar Askar) in branch '3.7': [3.7] bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891) (GH-8132)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ammar Askar : -- pull_requests: +7712 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ammar Askar : -- pull_requests: +7711 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ammar Askar : -- pull_requests: +7710 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Tal Einat added the comment: New changeset c4ef4896eac86a6759901c8546e26de4695a1389 by Tal Einat (Ammar Askar) in branch 'master': bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ammar Askar : -- keywords: +patch pull_requests: +7501 stage: -> patch review ___ Python tracker ___ ___

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

Change by Ned Deily : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

New submission from Ammar Askar : As was pointed out in https://bugs.python.org/issue33766 there is an edge case in the tokenizer whereby it will implicitly treat the end of input as a newline. The tokenize module in stdlib does not mirror the C code's behavior in this case. tokenizer.c: