[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-44297.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

Thanks, this fixes my issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 0acdf255a51b836c0b44f3676797620322974af3 by Mark Shannon in 
branch '3.10':
[3.10] bpo-43933: Force RETURN_VALUE bytecodes to have line numbers (GH-26061)
https://github.com/python/cpython/commit/0acdf255a51b836c0b44f3676797620322974af3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +24702
pull_request: https://github.com/python/cpython/pull/26061

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +24694
pull_request: https://github.com/python/cpython/pull/26054

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

BTW, I created a separate issue about this: https://bugs.python.org/issue44088 
because I wasn't sure if I should continue on this one, sorry.  Feel free to 
close either one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Change by Mark Shannon :


--
Removed message: https://bugs.python.org/msg393388

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

If there is no C-API function that supports your needs, feel free to suggest 
one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

Ned, no line numbers should never be None.
(Except under very unusual circumstances, like the trace function raising an 
error)

Taking a look at the disassembly of execsitecustomize, there is a return with 
no line number, which shouldn't happen.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

Mark, I'm not clear if the line number can still be None in some circumstances. 

With this code (from site.py in the stdlib):

545: def execsitecustomize():
546: """Run custom site specific code, if available."""
547: try:
548: try:
549: import sitecustomize
550: except ImportError as exc:
551: if exc.name == 'sitecustomize':
552: pass
553: else:
554: raise
555: except Exception as err:
556: if sys.flags.verbose:
557: sys.excepthook(*sys.exc_info())
558: else:
559: sys.stderr.write(
560: "Error in sitecustomize; set PYTHONVERBOSE for 
traceback:\n"
561: "%s: %s\n" %
562: (err.__class__.__name__, err))

I get traces with these events and line numbers:

exception 549 (ModuleNotFoundError("No module named 'sitecustomize'"))
line 550
line 551
line 552
return None

Is this what you expected?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Anthony Sottile]

Anthony Sottile  added the comment:

I've released pytest 6.2.4 for the other breakage (which is unrelated to this 
bpo -- it was the asname source position issue)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> The pytest failure still happens with 3.10.0b1 and pytest 6.2.3.

What I am missing? 3.10.0b1 includes the fixes for this issue, no? Is it 
because the fixes still require some changes in the source? If that's the case, 
we should mention it in the What's New

> I'm asking if pytest can make a release to update.

In any case, shouldn't that be requested in the pytest issue tracker?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

The pytest failure still happens with 3.10.0b1 and pytest 6.2.3.  I'm asking if 
pytest can make a release to update.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

>  Is this error going to happen for everyone testing with 3.10b1?

3.10.0b1 includes both of these PRs so people testing the beta should not get 
affected. That's why I marked this as release blocker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

Thanks, tip of pytest works for me.  Is this error going to happen for everyone 
testing with 3.10b1?  If so, can we get a release of pytest?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Anthony Sottile]

Anthony Sottile  added the comment:

ah yeah that's the `asname` ast change -- if you use the unreleased main branch 
it has a fix for that

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

I'm trying to see if these changes affect coverage.py, but pytest 6.2.3 fails 
with:

=== CPython 3.10.0a7+ (rev 558df90109) with Python tracer 
(.tox/anypy/bin/python) ===
ImportError while loading conftest 
'/Users/nedbatchelder/coverage/trunk/tests/conftest.py'.
TypeError: required field "lineno" missing from alias


Is this something I am doing wrong?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Anthony Sottile]

Anthony Sottile  added the comment:

can confirm that pytest no longer crashes

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Thanks Mark for the quick fix!

Can someone confirm if the pytest issue is fixed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

I'm claiming that this is fixed.

Feel free to reopen if the additional line in tracing causes a problem.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 5979e81a212949c62c2490167c9137d233d7de64 by Mark Shannon in 
branch 'master':
bpo-43933:  Set frame.f_lineno during call to __exit__ (GH-25719)
https://github.com/python/cpython/commit/5979e81a212949c62c2490167c9137d233d7de64


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:


New changeset 088a15c49d99ecb4c3bef93f8f40dd513c6cae3b by Mark Shannon in 
branch 'master':
bpo-43933: Show frame.f_lineno as None, rather than -1, if there is no line 
number. (GH-25717)
https://github.com/python/cpython/commit/088a15c49d99ecb4c3bef93f8f40dd513c6cae3b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +24410
pull_request: https://github.com/python/cpython/pull/25719

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

I'm going to set the line number for executing the `__exit__()` function as 
that of the `with` keyword.

This should give a sensible traceback, and not break coverage, as the with 
statement is already executed.
It will impact traces and profiles, which is why the __exit__ call was not 
given a line number before.

Anthony, Ned,
please let me know if you disagree.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Change by Mark Shannon :


--
keywords: +patch
pull_requests: +24408
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25717

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Petr Viktorin]

Petr Viktorin  added the comment:

Please don't lose the big picture here.
Previously, the reported line number was *somewhere close* to the currently 
executing code. That's typically all I need from a traceback: the fix for a bug 
often needs to go a few lines above/below where it's reported.

If, in an effort to make line numbers *exact*, we determine that there's no 
good line number to report and use -1 or None, we're introducing a regression.


> We could give it the line number of the `with` keyword, but that might break 
> coverage and profiling tools.

That sounds like a reasonable idea, despite the drawback. As we can see from 
the report, *any* change will probably break *some* tools.

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

This test case was added for 3.10.
In 3.9 it reports that the pass statement executes, even though it is 
unreachable.

https://github.com/python/cpython/blob/master/Lib/test/test_sys_settrace.py#L919

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

In this case, if happens that line 10 was the last line of code executing in 
the body of the with statement.
But the reason it was listed, was that it was statically the last line of code 
in the body.

If the body is something like:

1. if TRUE:
2. ...
3. else:
4. ...

Then the reported last line will be 4, even though it is never executed.
We can fix this, at the cost of extra compiler complexity and increased 
bytecode size for the non-exceptional case, but it is very difficult for the 
exceptional case, because we don't know statically where the exception has come 
from.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Ned Batchelder]

Ned Batchelder  added the comment:

Mark, you say:

> The traceback for 3.9 is incorrect, the program is not on line 10 when 
> __exit__ is called.

It's kind of not on any line in main(), but line 10 is the most useful line to 
indicate.  It's the last bit of the user's code that was running, and it is the 
reason for leaving the with statement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Mark Shannon]

Mark Shannon  added the comment:

What exactly is the issue here?
That the line number is -1, or that is different from 3.9?

The `-1` should be `None`. I can fix that, but there is the issue of whether 
the cleanup code in a with statement should have a line number or not.

The traceback for 3.9 is incorrect, the program is not on line 10 when __exit__ 
is called.

Currently we mark the cleanup code for the with statement as artificial meaning 
that it has no line number.
We could give it the line number of the `with` keyword, but that might break 
coverage and profiling tools.

PEP 626 says that the "with" keyword is executable, and should be traced, but 
nothing about whether that includes the cleanup call to __exit__.

--
nosy: +nedbat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Thanks for the bisection, Anthony

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Marking as release blocker since is breaking pytest

--
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
assignee:  -> Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43933] Regression in python3.10 with traceback frame having lineno of -1

[Anthony Sottile]

New submission from Anthony Sottile :

This is breaking pytest for failed assertions: 
https://github.com/pytest-dev/pytest/pull/8227

It also breaks the traceback in the output below

Here's a minimal example:

```python
class Boom:
def __enter__(self):
return self
def __exit__(self, *_):
raise AssertionError('boom!')


def main() -> int:
with Boom():
raise AssertionError('hi')


if __name__ == '__main__':
exit(main())
```

On python3.9 you get this:

```
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/t.py", line 10, in main
raise AssertionError('hi')
AssertionError: hi

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/t.py", line 14, in 
exit(main())
  File "/home/asottile/workspace/cpython/t.py", line 10, in main
raise AssertionError('hi')
  File "/home/asottile/workspace/cpython/t.py", line 5, in __exit__
raise AssertionError('boom!')
AssertionError: boom!
```

output in python3.10:

```
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/t.py", line 10, in main
raise AssertionError('hi')
AssertionError: hi

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/t.py", line 14, in 
exit(main())
  File "/home/asottile/workspace/cpython/t.py", line -1, in main
  File "/home/asottile/workspace/cpython/t.py", line 5, in __exit__
raise AssertionError('boom!')
AssertionError: boom!
```

Notice the second to last frame is now missing and invalid (line -1)

I bisected this and found this is the culprit:


https://github.com/python/cpython/commit/3bd6035b6baf1a7d51b7cc2c6bb2c81886236b67
https://github.com/python/cpython/pull/24202

```
3bd6035b6baf1a7d51b7cc2c6bb2c81886236b67 is the first bad commit
commit 3bd6035b6baf1a7d51b7cc2c6bb2c81886236b67
Author: Mark Shannon 
Date:   Wed Jan 13 12:05:43 2021 +

bpo-42908: Mark cleanup code at end of try-except and with artificial 
(#24202)

* Mark bytecodes at end of try-except as artificial.

* Make sure that the CFG is consistent throughout optimiization.

* Extend line-number propagation logic so that implicit returns after 
'try-except' or 'with' have the correct line numbers.

* Update importlib

 Lib/test/test_dis.py  |2 +-
 Lib/test/test_sys_settrace.py |   40 +
 Python/compile.c  |  135 +-
 Python/importlib.h| 3153 ++---
 Python/importlib_external.h   | 4489 -
 Python/importlib_zipimport.h  | 1013 +-
 6 files changed, 4473 insertions(+), 4359 deletions(-)
bisect run success
```

which appears to be due to bpo-42908

--
messages: 391795
nosy: Anthony Sottile, Mark.Shannon, pablogsal
priority: normal
severity: normal
status: open
title: Regression in python3.10 with traceback frame having lineno of -1
versions: Python 3.10, Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com