[issue25324] Importing tokenize modifies token

2017-06-06 Thread Serhiy Storchaka 

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed
versions: +Python 3.7 -Python 2.7, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-06 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:


New changeset 5cefb6cfdd089d237ba6724bb5311ee4f04be59f by Serhiy Storchaka in 
branch 'master':
bpo-25324: Move the description of tokenize tokens to token.rst. (#1911)
https://github.com/python/cpython/commit/5cefb6cfdd089d237ba6724bb5311ee4f04be59f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread Serhiy Storchaka 

Changes by Serhiy Storchaka :


--
pull_requests: +1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread STINNER Victor 

STINNER Victor added the comment:

Albert-Jan Nijburg added the comment:
> Aah! Oops I can fix later today.

Don't worry. Our test suite and all reviewers also missed it ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread Mariatta Wijaya 

Mariatta Wijaya added the comment:


New changeset c9ccacea3ff441b1ff519c7399602b7db16f9783 by Mariatta (Albert-Jan 
Nijburg) in branch 'master':
bpo-25324: add missing comma in Parser/tokenizer.c (GH-1910)
https://github.com/python/cpython/commit/c9ccacea3ff441b1ff519c7399602b7db16f9783


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread Albert-Jan Nijburg 

Changes by Albert-Jan Nijburg :


--
pull_requests: +1990

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

Aah! Oops I can fix later today.

On Thu, 1 Jun 2017 at 18:08, STINNER Victor  wrote:

>
> STINNER Victor added the comment:
>
> We got a bug report from Coverity:
>
> *** CID 1411801:  Incorrect expression  (MISSING_COMMA)
> /Parser/tokenizer.c: 111 in ()
> 105 "OP",
> 106 "AWAIT",
> 107 "ASYNC",
> 108 "",
> 109 "COMMENT",
> 110 "NL",
> >>> CID 1411801:  Incorrect expression  (MISSING_COMMA)
> >>> In the initialization of "_PyParser_TokenNames", a suspicious
> concatenated string ""ENCODING"" is produced.
> 111 "ENCODING"
> 112 ""
> 113 };
> 114
> 115
> 116 /* Create and initialize a new tok_state structure */
>
> I missed this typo :-p
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-06-01 Thread STINNER Victor 

STINNER Victor added the comment:

We got a bug report from Coverity:

*** CID 1411801:  Incorrect expression  (MISSING_COMMA)
/Parser/tokenizer.c: 111 in ()
105 "OP",
106 "AWAIT",
107 "ASYNC",
108 "",
109 "COMMENT",
110 "NL",
>>> CID 1411801:  Incorrect expression  (MISSING_COMMA)
>>> In the initialization of "_PyParser_TokenNames", a suspicious 
>>> concatenated string ""ENCODING"" is produced.
111 "ENCODING"
112 ""
113 };
114
115
116 /* Create and initialize a new tok_state structure */

I missed this typo :-p

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-31 Thread STINNER Victor 

STINNER Victor added the comment:


New changeset fc354f07855a9197e71f851ad930cbf5652f9160 by Victor Stinner 
(Albert-Jan Nijburg) in branch 'master':
bpo-25324: copy tok_name before changing it (#1608)
https://github.com/python/cpython/commit/fc354f07855a9197e71f851ad930cbf5652f9160


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-31 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

If move COMMENT, NL and ENCODING to the token module, their documentation 
should be moved from tokenize.rst to token.rst.

I have asked on Python-Dev whether this is a right way. [1]

[1] https://mail.python.org/pipermail/python-dev/2017-May/148080.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-31 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

I've updated token.rst and Misc/NEWS. Let me know if the wording is correct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-30 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

If merge this only in master, the new members of the token module should be 
documented (add a versionchanged directive in token.rst). This also needs a 
Misc/NEWS entry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-30 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

Let me know if you want me to add/change anything about my PR :) I'm happy to 
do so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-30 Thread STINNER Victor 

STINNER Victor added the comment:

I suggest to only modify the master branch. Nobody complained before. It's only 
a very minor surprising behaviour.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-30 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Albert-Jan's patch should be merged prior my patch for issue30455, because it 
can be backported, by my patch shouldn't. But first we should compare them and 
find differences in results.

My patch adds COMMENT, NL and ENCODING in the documentation, Albert-Jan's patch 
doesn't. I don't know whether it is worth to add these names to the token 
module in bugfix releases, and whether it is worth to document them.

My patch wraps all names in _PyParser_TokenNames after  with angle 
parenthesis, Albert-Jan's patch inserts new names between  and 
 without angle parenthesis. I don't know what is more correct.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-30 Thread STINNER Victor 

STINNER Victor added the comment:

https://github.com/python/cpython/pull/1608 doesn't include a Misc/NEWS entry, 
but I don't want to stress albertjan with Misc/NEWS since I was already 
annoying enough on the review :-) I'm not sure that the change deserves a NEWS 
entry. If you consider that yes, it does. It can be done in a separated change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-24 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

I've updated the PR and added the tokenize tokens to token.h and their names to 
tokenizer.c. This way they'll show up when you run token.py. The names will 
always be in tok_name and tokenizer.py will use those. Not breaking the public 
api and no longer modifying token.py when you import tokenizer.py.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread STINNER Victor 

STINNER Victor added the comment:

> lib2to3 appears to have it's own token.py as well with NL and COMMENT 
> withtout ENCODING...

Oh you are right: Lib/lib2to3/pgen2/driver.py uses Lib/lib2to3/pgen2/token.py.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

lib2to3 appears to have it's own token.py as well with NL and COMMENT withtout 
ENCODING... 

Lib/lib2to3/pgen2/token.py

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread Albert-Jan Nijburg 

Albert-Jan Nijburg added the comment:

> I prefer to add tokenize tokens directly in Lib/token.py, and then get 
> COMMENT, NL and ENCODING using tok_name.index().

That would make more sense from a breaking change perspective, but we would 
step on the toes of anyone adding `COMMENT`, `NL`, or `ENCODING` to `token.h` 
because `token.py` is generated from that.

It would also make much more sense to have them as fields on `token` if they 
are in `tok_name` in `token`.

--
nosy: +Albert-Jan Nijburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread STINNER Victor 

STINNER Victor added the comment:

Another example from Tools/i18n/pygettext.py, TokenEater::

def __call__(self, ttype, tstring, stup, etup, line):
# dispatch
##import token
##print >> sys.stderr, 'ttype:', token.tok_name[ttype], \
##  'tstring:', tstring
self.__state(ttype, tstring, stup[0])


...

eater.set_filename(filename)
try:
tokens = tokenize.tokenize(fp.readline)
for _token in tokens:
eater(*_token)

Another example using token.tok_name with token types coming from tokenize.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread STINNER Victor 

STINNER Victor added the comment:

Extract of Lib/lib2to3/pgen2/driver.py:

if type in (tokenize.COMMENT, tokenize.NL):
...
if debug:
self.logger.debug("%s %r (prefix=%r)",
  token.tok_name[type], value, prefix)

The code uses tokenize.COMMENT and look for this constant into token.tok_name. 
If token.tok_name doesn't contain COMMENT anymore, it breaks lib2to3, no? At 
least the debug mode which might not be covered by test_lib2to3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread STINNER Victor 

STINNER Victor added the comment:

> I would fix this by making tokenize.tok_name a copy. It looks like this 
> behaviour dates back to 1997 (see revision 1efc4273fdb7).

token.tok_name is part of the Python public API:
https://docs.python.org/dev/library/token.html#token.tok_name

whereas tokenize.tok_name isn't documented. So I dislike having two 
disconnected mappings. I prefer to add tokenize tokens directly in 
Lib/token.py, and then get COMMENT, NL and ENCODING using tok_name.index().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2017-05-16 Thread Albert-Jan Nijburg 

Changes by Albert-Jan Nijburg :


--
pull_requests: +1699

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2015-12-22 Thread Joseph Gordon 

Changes by Joseph Gordon :


--
nosy: +josephgordon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2015-10-06 Thread Serhiy Storchaka 

New submission from Serhiy Storchaka:

Importing the tokenize modifies the content of the token module.

>>> import token
>>> len(token.tok_name)
59
>>> import tokenize
>>> len(token.tok_name)
61

Such side effect looks as bad design to me. Either token.tok_name should 
contain elements needed for tokenize, or tokenize should not modify 
token.tok_name.

--
components: Library (Lib)
messages: 252397
nosy: georg.brandl, meador.inge, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Importing tokenize modifies token
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2015-10-06 Thread STINNER Victor 

STINNER Victor added the comment:

I agree :-) Do you want to work on a patch?

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2015-10-06 Thread Martin Panter 

Martin Panter added the comment:

I would fix this by making tokenize.tok_name a copy. It looks like this 
behaviour dates back to 1997 (see revision 1efc4273fdb7).

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25324] Importing tokenize modifies token

2015-10-06 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

I left the decision for modules maintainers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com