[issue42687] tokenize module does not recognize Barry as FLUFL

[Erik Soma]

Change by Erik Soma :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42687] tokenize module does not recognize Barry as FLUFL

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

I concur with Terry.

--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42687] tokenize module does not recognize Barry as FLUFL

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I strongly disagree.  '<>' is not a legal operator any more.  It is a 
parse-time syntax error.  Whatever historical artifact is left in the CPython 
tokenizer, recognizing '<>' is not exposed to Python code.

>>> p = ast.parse('a <> b')
Traceback (most recent call last):
...
a <> b
^
SyntaxError: invalid syntax  

When '<>' was legal, we may presume that tokenizer recognized it, so that not 
recognizing it was an intentional change.  Reverting this would be a 
dis-service to users.  

I think that the PR and this issue should be closed.  If the historical 
artifact bothers you, propose removing it instead on introducing a bug into 
tokenizer.

--
nosy: +terry.reedy
type:  -> enhancement
versions:  -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42687] tokenize module does not recognize Barry as FLUFL

[Erik Soma]

Change by Erik Soma :


--
keywords: +patch
pull_requests: +22722
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/23857

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42687] tokenize module does not recognize Barry as FLUFL

[Erik Soma]

New submission from Erik Soma :

'<>' is not recognized by the tokenize module as a single token, instead it is 
two tokens.

```
$ python -c "import tokenize; import io; import pprint; 
pprint.pprint(list(tokenize.tokenize(io.BytesIO(b'<>').readline)))"
[TokenInfo(type=62 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), 
line=''),
 TokenInfo(type=54 (OP), string='<', start=(1, 0), end=(1, 1), line='<>'),
 TokenInfo(type=54 (OP), string='>', start=(1, 1), end=(1, 2), line='<>'),
 TokenInfo(type=4 (NEWLINE), string='', start=(1, 2), end=(1, 3), line=''),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
```


I would expect:
```
[TokenInfo(type=62 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), 
line=''),
 TokenInfo(type=54 (OP), string='<>', start=(1, 0), end=(1, 2), line='<>'),
 TokenInfo(type=4 (NEWLINE), string='', start=(1, 2), end=(1, 3), line=''),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
```

This is the behavior of the CPython tokenizer which the tokenizer module tries 
"to match the working of".

--
messages: 383384
nosy: esoma
priority: normal
severity: normal
status: open
title: tokenize module does not recognize Barry as FLUFL
versions: Python 3.10, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com