[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[miss-islington]

miss-islington  added the comment:


New changeset efd878cdb46d9c7038d93fb36eb1ff7dc5baf9ec by Miss Islington (bot) 
in branch '3.8':
bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)
https://github.com/python/cpython/commit/efd878cdb46d9c7038d93fb36eb1ff7dc5baf9ec


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 0cc6b5e559b8303b18fdd56c2befd900fe7b5e35 by Serhiy Storchaka in 
branch 'master':
bpo-39219: Fix SyntaxError attributes in the tokenizer. (GH-17828)
https://github.com/python/cpython/commit/0cc6b5e559b8303b18fdd56c2befd900fe7b5e35


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[miss-islington]

Change by miss-islington :


--
pull_requests: +17849
pull_request: https://github.com/python/cpython/pull/18479

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Sorry, wrong issue number.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +17360
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17828

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Terry J. Reedy]

Terry J. Reedy  added the comment:

On entry of '0xz', IDLE from 3.6 to date highlights the '0x' part of the 
original entry.  I presume it can do this because it ignores the text 
attribute, and because bytes == chars for at least  '0x'.

The 'proposed PR' is not listed here.  Is it not yet submitted, or is the issue 
number wrong?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

SyntaxError can be raised at different stages of compiling. In some places the 
source text is not available and should be read from the file using the file 
name. Which does not work in case of compiling a string or reading from stdin.

>>> 0z
  File "", line 1
0z
 ^
SyntaxError: invalid syntax
>>> 0xz
  File "", line 1
SyntaxError: invalid hexadecimal literal

In the second example above the source line and the caret are absent in REPL.

The proposed PR fixes two errors in raising a SyntaxError in the tokenizer.

1. The text of the source line was not set if an exception was raised in the 
tokenizer. Since most of these exceptions (with more detailed description) were 
added in 3.8 I consider this a regressions.

2. The offset attribute was an offset in bytes. Now it is an offset in 
characters.

It only fixes errors in the tokenizer. There are similar bugs in other parts of 
the compiler.

This issue is based on the article 
https://aroberge.blogspot.com/2019/12/a-tiny-python-exception-oddity.html .

--
components: Interpreter Core
messages: 359331
nosy: serhiy.storchaka
priority: normal
severity: normal
status: open
title: Fix attributes of syntax errors raized in the tokenizer
type: behavior
versions: Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com