[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 


Pablo Galindo Salgado  added the comment:


New changeset bfc413ce4fa37ccb889757388102c7755e057bf5 by Pablo Galindo in 
branch '3.9':
[3.9] bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067) 
(GH-24069)
https://github.com/python/cpython/commit/bfc413ce4fa37ccb889757388102c7755e057bf5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 


Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 


Change by Pablo Galindo Salgado :


--
pull_requests: +22902
pull_request: https://github.com/python/cpython/pull/24069

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 


Pablo Galindo Salgado  added the comment:


New changeset bd2728b1e8a99ba8f8c2d481f88aeb99b8b8360f by Pablo Galindo in 
branch 'master':
bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067)
https://github.com/python/cpython/commit/bd2728b1e8a99ba8f8c2d481f88aeb99b8b8360f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 

Pablo Galindo Salgado  added the comment:

Seems that I also found this weird behaviour in the old parser:

import ast
code = """\
x = (
'PERL_MM_OPT', (
f'wat'
f'INSTALL-BASE={shlex.quote(venv)} '
f'wat'
),
)
"""
elem = ast.parse(code).body[0].value.elts[1].values[1].value
print(elem)
print(code.split("\n")[elem.lineno-1][elem.col_offset:elem.end_col_offset])

In Python3.8 this prints:

<_ast.Call object at 0x7f7484393a00>
'wat'

which is wrong as the code for that call is certainly not "wat". This happens 
in the oldparser in 3.9:

❯ ../3.9/python -Xoldparser lel.py

'wat'

And something wrong happens in the current master with the new parser:

❯ ../3.9/python lel.py

STALL-BASE={shlex

But with PR24067:


shlex.quote(venv)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Pablo Galindo Salgado 


Change by Pablo Galindo Salgado :


--
keywords: +patch
pull_requests: +22900
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24067

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-02 Thread Anthony Sottile 


Anthony Sottile  added the comment:

by the way, here's a much smaller case which has a similar problem

```
x = (
f" {test(t)}"
)
```

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-01 Thread Anthony Sottile 


Anthony Sottile  added the comment:

I didn't initially, but now I have and it has the same bug in the current 
master: 3bf05327c2b25d42b92795d9d280288c22a0963d

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-01 Thread Guido van Rossum 


Guido van Rossum  added the comment:

Did you check this with master?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-01 Thread Anthony Sottile 


Anthony Sottile  added the comment:

+peg parser individuals to nosy

--
nosy: +gvanrossum, lys.nikolaou, pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42806] Incorrect offsets in new parser for f-string substitutions

2021-01-01 Thread Anthony Sottile 


New submission from Anthony Sottile :

here's a sample file.  this is distilled down from an example which broke a 
code formatter here: https://github.com/asottile/add-trailing-comma/issues/106

def f():
x = (
'PERL_MM_OPT', (
f'INSTALL-BASE={shlex.quote(venv)} '
f'wat'
),
)


A diff from astpretty (https://github.com/asottile/astpretty) between the old 
and new parser:

```console
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) 
<(.tox/py39/bin/python -m astpretty perl.py)
--- /dev/fd/63  2021-01-01 15:58:09.108060968 -0800
+++ /dev/fd/62  2021-01-01 15:58:09.108060968 -0800
@@ -35,19 +35,19 @@
 end_col_offset=22,
 value=Call(
 lineno=4,
-col_offset=32,
+col_offset=16,
 end_lineno=4,
-end_col_offset=49,
+end_col_offset=33,
 func=Attribute(
 lineno=4,
-col_offset=32,
+col_offset=16,
 end_lineno=4,
-end_col_offset=43,
-value=Name(lineno=4, 
col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+end_col_offset=27,
+value=Name(lineno=4, 
col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
 attr='quote',
 ctx=Load(),
 ),
-args=[Name(lineno=4, 
col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+args=[Name(lineno=4, 
col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
 keywords=[],
 ),
 conversion=-1,
```

the old parser is correct here, and the new parser is wrong

notably it thinks that the call inside the f-string substitution is positioned 
at column 16 which is the position of the f-string start

this bug is also present in 3.10:

```console
$ venv/bin/python3.10 --version
Python 3.10.0a3
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) 
<(venv/bin/python3.10 -m astpretty perl.py)
--- /dev/fd/63  2021-01-01 15:59:54.59968 -0800
+++ /dev/fd/62  2021-01-01 15:59:54.59968 -0800
@@ -35,19 +35,19 @@
 end_col_offset=22,
 value=Call(
 lineno=4,
-col_offset=32,
+col_offset=16,
 end_lineno=4,
-end_col_offset=49,
+end_col_offset=33,
 func=Attribute(
 lineno=4,
-col_offset=32,
+col_offset=16,
 end_lineno=4,
-end_col_offset=43,
-value=Name(lineno=4, 
col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+end_col_offset=27,
+value=Name(lineno=4, 
col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
 attr='quote',
 ctx=Load(),
 ),
-args=[Name(lineno=4, 
col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+args=[Name(lineno=4, 
col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
 keywords=[],
 ),
 conversion=-1,
```

--
components: Interpreter Core
messages: 384208
nosy: Anthony Sottile
priority: normal
severity: normal
status: open