[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Changes by Mariatta Wijaya :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Mariatta Wijaya added the comment:


New changeset ea0f7c26cef4550bf4db1a9bae17d41b79ab7c0d by Mariatta in branch 
'master':
bpo-31394: Make tokenize.rst PEP 8-compliant (GH-3526)
https://github.com/python/cpython/commit/ea0f7c26cef4550bf4db1a9bae17d41b79ab7c0d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Changes by Mariatta Wijaya :


--
pull_requests: +3526

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Mariatta Wijaya added the comment:


New changeset 5513e888e9a742156c35ce7ab628407d8cf9e1f0 by Mariatta in branch 
'3.6':
[3.6] bpo-31394: Clarify documentation about token type attribute (GH-3469) 
(GH-3525)
https://github.com/python/cpython/commit/5513e888e9a742156c35ce7ab628407d8cf9e1f0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Changes by Mariatta Wijaya :


--
pull_requests: +3525

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-12 Thread Mariatta Wijaya 

Mariatta Wijaya added the comment:


New changeset 5f8fbf917ebf2398aa75a1f271617e2e50ab7c88 by Mariatta (Aivar 
Annamaa) in branch 'master':
bpo-31394: Clarify documentation about token type attribute (GH-3469)
https://github.com/python/cpython/commit/5f8fbf917ebf2398aa75a1f271617e2e50ab7c88


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-09 Thread Terry J. Reedy 

Terry J. Reedy added the comment:

Once the CLA * shows up after your name, one of us should remove CLA needed tag 
and bot will verify.  I believe tagging as trivial would also work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-09 Thread Aivar Annamaa 

Aivar Annamaa added the comment:

Here is the PR: https://github.com/python/cpython/pull/3469

(It's my first, so I don't know if I should to also update NEWS file or add 
"skip news" label. I signed the CLA, so I hope this warning goes away.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-09 Thread Roundup Robot 

Changes by Roundup Robot :


--
keywords: +patch
pull_requests: +3461
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-08 Thread Berker Peksag 

Berker Peksag added the comment:

Thank you for your detailed response, Aivar. I agree that adding 'and Ellipsis' 
would make the tokenize documentation clearer. Would you like to send a pull 
request?

--
keywords: +easy
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-08 Thread Terry J. Reedy 

Terry J. Reedy added the comment:

I think we should add 'and Ellipsis' in the tokenize doc to get
'all Operators and Delimiters tokens and Ellipsis are returned ...'.  I would 
actually prefer 'Operator and Delimiter tokens' but I don't know if the 's' is 
needed to trigger the current linkage.

--
assignee:  -> docs@python
components: +Documentation -Interpreter Core
nosy: +docs@python, terry.reedy
stage: resolved -> needs patch
versions: +Python 3.7 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-08 Thread Aivar Annamaa 

Aivar Annamaa added the comment:

(Sorry, I didn't mean to challenge the authority of a core developer. I simply 
didn't notice that adding a comment reopens the issue. I hope this time I 
selected correct parameters and this doesn't happen again)

I'm trying to rephrase my concern. 

Initially I thought there was a mistake in the tokenizer or in the token module.

After you pointed out the documentation about token.OP and exact_type, I'm 
worried about a smaller detail. The documentation only talks about *operators* 
and *delimiters* having type attribute set to token.OP. According to my 
understanding (and also the listings at 
https://docs.python.org/3/reference/lexical_analysis.html#operators), ellipsis 
is neither operator nor delimiter. 

Am I right about this? 

(I understand that the source representation of both ELLIPSIS and DOT tokens 
contains period *character(s)*, but I don't see why is this relevant when we 
discuss the properties of *tokens*)

Anyway, if ellipsis is neither operator nor delimiter, (and if there is a 
reason why it is treated similarly with operators and delimiters) then I 
recommend to update the documentation by replacing 

> all Operators and Delimiters tokens 
> are returned using the generic token.OP token type

with 

> all Operators, Delimiters and Ellipsis tokens
> are returned using the generic token.OP token type


(I understand, that this is not a serious issue. If you prefer not to discuss 
it further then I'm happy to leave it as it is.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-08 Thread Berker Peksag 

Berker Peksag added the comment:

Please don't reopen an issue if it was closed by a core developer.

It's not clear to me what exactly do you want to change in the implementation 
or documentation.

 A sequence of three periods has a special meaning as an ellipsis literal.

is literally describes how the ELLIPSIS token is identified in 
Parser/tokenizer.c (see PyToken_ThreeChars in that file) So a sequence of three 
periods is identified as an ellipsis literal which is an expression in Python.

Do you want to change tokenize.tokenize() (it's in Lib/tokenize.py) so it will 
return

TokenInfo(type=52 (ELLIPSIS), ...)

instead of

TokenInfo(type=53 (OP), ...)

? Note that ELLIPSIS has been added to tokenize.EXACT_TOKEN_TYPES in issue 
24622. To me, since it has been added to tokenize.EXACT_TOKEN_TYPES there is no 
need to special case ELLIPSIS in Lib/tokenize.py.

Or do you want to clarify

To simplify token stream handling, all Operators and Delimiters tokens are 
returned using the generic token.OP token type.

at https://docs.python.org/3/library/tokenize.html?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31394] Ellipsis_token.type != token.ELLIPSIS

2017-09-08 Thread Aivar Annamaa 

Aivar Annamaa added the comment:

But ellipsis is a distinct token, not a sequence of three period tokens.

Also, I can't see how we could conceptually treat ellipsis as a delimiter or 
operator -- it's a literal.

I still think either documentation or implementation needs to be fixed here.

--
resolution: not a bug -> 
status: closed -> open
title: Ellipsis token.type != token.ELLIPSIS -> Ellipsis_token.type != 
token.ELLIPSIS
type: behavior -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com