[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2018-04-22 Thread Ned Batchelder
Ned Batchelder added the comment: PR 6273 is mentioned, but I think 6573 is the correct number. -- ___ Python tracker ___ ___ Python

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2018-04-22 Thread Łukasz Langa
Change by Łukasz Langa : -- pull_requests: +6273 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-11-03 Thread Ezio Melotti
Ezio Melotti added the comment: Fixed, thanks for the report and the patch! -- assignee: -> ezio.melotti nosy: +ezio.melotti resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed type: -> behavior versions: +Python 3.2 _

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-11-03 Thread Roundup Robot
Roundup Robot added the comment: New changeset eb7ea51e658e by Ezio Melotti in branch '2.7': #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. http://hg.python.org/cpython/rev/eb7ea51e658e New changeset 31798ed5 by Ezio Melo

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-09 Thread Ned Batchelder
Ned Batchelder added the comment: Jesús, done! -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.p

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-09 Thread Jesús Cea Avión
Changes by Jesús Cea Avión : -- versions: +Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-09 Thread Jesús Cea Avión
Jesús Cea Avión added the comment: Ned, could you possibly send a Contributor Form Agreement? http://www.python.org/psf/contrib/ -- nosy: +jcea ___ Python tracker ___ __

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
Ned Batchelder added the comment: Updated with new (better) patch, for v2.7 and v3.3. They are the same except for the test. -- ___ Python tracker ___ _

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
Changes by Ned Batchelder : Added file: http://bugs.python.org/file27467/bug16152_v27.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
Changes by Ned Batchelder : Added file: http://bugs.python.org/file27466/bug16152_v33.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
Changes by Ned Batchelder : Removed file: http://bugs.python.org/file27459/bug16152.patch ___ Python tracker ___ ___ Python-bugs-list mailing

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
Ned Batchelder added the comment: Here's a patch for 3.3. I would like to also fix 2.7... -- keywords: +patch Added file: http://bugs.python.org/file27459/bug16152.patch ___ Python tracker

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread STINNER Victor
Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue16152] Trailing whitespace makes tokenize.generate_tokens pathological

2012-10-06 Thread Ned Batchelder
New submission from Ned Batchelder: When tokenizing with tokenize.generate_tokens, if the code ends with whitespace (no newline), the tokenizer produces an ERRORTOKEN for each space. Additionally, the regex that fails to find tokens in those spaces is linear in the number of spaces, so the ov