[issue40222] "Zero cost" exception handling

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-47236 "Document types.CodeType.replace() changes about 
co_exceptiontable".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

> See bpo-47185: code.replace(co_code=new_code) no longer catch exceptions on 
> Python 3.11.

Surely the bigger issue is that the contents of new_code itself must be totally 
different? Also there are other tables that need to be adjusted if you really 
do change co_code, e.g. the debugging tables.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[STINNER Victor]

STINNER Victor  added the comment:

See bpo-47185: code.replace(co_code=new_code) no longer catch exceptions on 
Python 3.11.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

Yes.

--
priority: release blocker -> 
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Can this be closed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Irit Katriel]

Irit Katriel  added the comment:


New changeset 7989e9dff68fd681657aa9f93ac6e8fa12c534e4 by Irit Katriel in 
branch 'main':
bpo-40222: update doc entry with respect to the change in WITH_EXCEPT_START 
(GH-29975)
https://github.com/python/cpython/commit/7989e9dff68fd681657aa9f93ac6e8fa12c534e4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +iritkatriel
nosy_count: 16.0 -> 17.0
pull_requests: +28199
pull_request: https://github.com/python/cpython/pull/29975

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Irit Katriel]

Change by Irit Katriel :


--
nosy: +iritkatriel, iritkatriel
nosy_count: 16.0 -> 17.0
pull_requests: +28199, 28200
pull_request: https://github.com/python/cpython/pull/29975

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Ruairidh MacLeod]

Change by Ruairidh MacLeod :


--
nosy: +rkm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

I've started a thread on python-dev.

https://mail.python.org/archives/list/python-...@python.org/thread/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/#ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

> >> I want to get rid of PyCode_NewWithPosArgs() and just have PyCode_New(). 

> That as added because of PEP 387 and unfortunately removing it is backwards 
> incompatible.

Is changing the signature allowed? Because it *must* be changed (at the very 
least to accommodate the exceptiontable, but there are several others too -- 
your PEP 657 touched it last to add endlinetable and columntable).

I think this was a mistake in PEP 387 and we just need to retract that. Perhaps 
it could be left as a dummy that always returns an error?

> >> I propose we declare all APIs for code objects *unstable*, liable to 
> >> change each (feature) release.

> I agree that we should do this, but this needs at least a discussion in 
> python-dev because currently these APIs are protected by PEP 387 so changing 
> them is backwards incompatible

Yeah that's the crux. :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

>> I want to get rid of PyCode_NewWithPosArgs() and just have PyCode_New(). 

That as added because of PEP 387 and unfortunately removing it is backwards 
incompatible.

>> I propose we declare all APIs for code objects *unstable*, liable to change 
>> each (feature) release.

I agree that we should do this, but this needs at least a discussion in 
python-dev because currently these APIs are protected by PEP 387 so changing 
them is backwards incompatible

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

I propose we declare all APIs for code objects *unstable*, liable to change 
each (feature) release.

I want to get rid of PyCode_NewWithPosArgs() and just have PyCode_New(). All 
callers to either one must be changed anyways. (And we’re not done changing 
this in 3.11 either.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> Is there a separate issue for how we are handling CodeType()?

No, that's why this is marked as release blocker, because this is the first 
issue where CodeType was changed. 

If you wish to close this one, please open a new issue explaining the situation 
and mark that one as release blocker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

I'd like to close this, as the exception handling is all done and working 
correctly.

Is there a separate issue for how we are handling CodeType()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

Ah, okay. So we're not on the hook to decide this today. :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Small note, as I mentioned in my correction email 
(https://mail.python.org/archives/list/python-...@python.org/message/GBD57CUUU4K5NMQDTEZXNCX76YISEIGQ/),
 this is a release blocker for 3.11 (it was not marked in the issue what Python 
version was associated, I am doing it with this message) so this doesn't block 
the 3.10 release.

--
versions: +Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I think we're waiting here for the release manager to decide, right?

As Petr mentions, the release manager doesn't have authority to decide if the 
backwards compatibility policy can be ignored, only the Steering Council.

> Should we roll back the change to PyCode_NewWithPosOnlyArgs() or keep it?

I don't think is possible: code objects must be constructed with the new 
argument, otherwise they are broken. There is not an easy way to have a default 
for PyCode_New and PyCode_NewWithPosOnlyArgs that somehow creates the field 
from nothing. 

I *personally* think that this case is one good example of an exception to the 
backwards compact rule, but I myself cannot grant that exception as a release 
manager. I also think these APIs should be removed from the public C-API ASAP 
because they literally conflict everytime we change the code object for 
optimizations.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

I think we're waiting here for the release manager to decide, right? Should we 
roll back the change to PyCode_NewWithPosOnlyArgs() or keep it?

Presumably the requested docs aren't the (beta) release blocker?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[STINNER Victor]

STINNER Victor  added the comment:

Mark: Can you please document your change on types.CodeType? In:

* https://docs.python.org/dev/library/types.html#types.CodeType
* https://docs.python.org/dev/whatsnew/3.11.html

The change broke the Genshi project:
https://github.com/edgewall/genshi/issues/43

in this Genshi function:

def build_code_chunk(code, filename, name, lineno):
params =  [0, code.co_nlocals, code.co_kwonlyargcount,
  code.co_stacksize, code.co_flags | 0x0040,
  code.co_code, code.co_consts, code.co_names,
  code.co_varnames, filename, name, lineno,
  code.co_lnotab, (), ()]
if hasattr(code, "co_posonlyargcount"):
# PEP 570 added "positional only arguments"
params.insert(2, code.co_posonlyargcount)
return CodeType(*params)

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +24699
pull_request: https://github.com/python/cpython/pull/26059

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Stefan Behnel]

Stefan Behnel  added the comment:

Just a comment regarding the change to "PyCode_NewWithPosOnlyArgs()". As Pablo 
mentioned, this has happened before. And that's OK! Exactly because this has 
happened before, it's clearly not a part of the API that is meant to be stable.

I can easily adapt Cython to make this work in the next patch-level release of 
CPython 3.11 (or the current one, since alpha-1 seems not so close), but any 
adaptation will be patch-level dependent. Meaning, for each such change, there 
will be a couple of weeks or months until the C preprocessor makes the code 
compile again. And during that time, people won't be able to test their code to 
report issues.

So, I'd rather have compatibility broken and stay that way, than going one way 
now and changing it back later, thus going through the same adaptation period 
twice.

That being said, any such change means that maintainers will have to rebuild 
their packages with a new Cython release to adapt them to Py3.11. Many will, 
but some won't, for whatever reason.

--
nosy: +scoder

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> It is very little effort to add back the old function, so that isn't the 
> problem. It won't work properly, but it never did anyway. So I guess that's 
> sort of compatible.

It won't work properly is an incompatible change. Before, if you extract all 
fields from a code object and pass it down to the constructor, everything will 
work.

> Maybe the best thing is to put a big red warning in the docs and hope that 
> warns away people from using it?

I think code object constructors must be part of the private CAPI due to what 
we are experiencing. But again, this is something we cannot decide on this bpo 
issue. Either a python-dev thread needs to be open or a Steering Council 
request in the https://github.com/python/steering-council repo needs to be 
opened.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

It is very little effort to add back the old function, so that isn't the 
problem. It won't work properly, but it never did anyway. So I guess that's 
sort of compatible.

Maybe the best thing is to put a big red warning in the docs and hope that 
warns away people from using it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> I think API breaks like this do need coordination at the project level.

Absolutely, that's why I said before:

> As I mentioned, I simphatise with your argument and I think it makes 
 sense, but we cannot just do it in a BPO issue, I'm afraid.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Petr Viktorin]

Petr Viktorin  added the comment:

Then, according to PEP 387, "The steering council may grant exceptions to this 
policy."

I think API breaks like this do need coordination at the project level.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> The PyAPI_FUNC makes it part of the public *API*. It needs to be source- 
> compatible; the number of arguments can't change. Could yo u add a new 
> function?

Unfortunately, no: new functions cannot be added easily because the new field 
that receives is needed and is a complicated field created by the compiler. The 
old API is not enough anymore and making a compatibility layer is a huge 
complexity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Petr Viktorin]

Petr Viktorin  added the comment:

PyCode_NewWithPosOnlyArgs is not part of the stable ABI. It is OK to break its 
ABI in a minor version (i.e. 3.11).

The PyAPI_FUNC makes it part of the public *API*. It needs to be source- 
compatible; the number of arguments can't change. Could yo u add a new function?

I wouldn't remove PyCode_NewWithPosOnlyArgs from the public C API, which can be 
CPython-specific and used by projects like Cython that need some low-level 
access for performance. But PEP 387 applies, so if it is deprecated in 3.11, it 
can be removed in 3.13.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

In any case, if we decide to let it stay, at the very least this behaviour 
(that these functions are not stable) needs to be documented everywhere: what's 
new, C-API docs...etc And probably we need to somehow add it to the future 
deprecations of 3.10 for visibility.

As I mentioned, I simphatise with your argument and I think it makes sense, but 
we cannot just do it in a BPO issue, I'm afraid.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

I agree with you but we already went through this when I added positional-only 
arguments and everyone complained that Cython and other projects were broken 
and we changed a stable API function so I am just mentioning that we are here 
again.

Honestly, I think what you describe makes sense and this constructor should 
never be stable (as the Python one is not stable). If we add column offsets 
that's another parameter more than we would need to add. But in any case that's 
just my opinion and we should reach some conclusion collectively, that's why I 
mentioned this here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

I know PyCode_NewWithPosOnlyArgs is declared as "PyAPI_FUNC" but that can't 
make it part of the ABI unless it has stable behavior.
It can't have stable behavior because its inputs are complex, undefined, have 
altered semantics and are interlinked in complex ways.

Passing the same arguments to PyCode_NewWithPosOnlyArgs for both 3.9 and 3.10 
will cause one or other version to crash (interpreter crash, not just program 
crash).


We need to stop adding "PyAPI_FUNC" to everything.
Adding a PyAPI_FUNC does not magically make for ABI compatibility, there is a 
lot more to it than that.

The only sane ways to construct a code object are to load it from disk, to 
compile an AST, or to use
codeobject.replace(). Any purported ABI compatibility claims are just 
misleading and a trap.


I can revert the API changes and add a new function, but I think that is 
dangerously misleading. A compilation error is preferable to an interpreter 
crash.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

It seems that we have broken the stable ABI of PyCode_NewWithPosOnlyArgs as it 
has now another parameter. We need to either create a new private constructor 
or a new public constructor, but the ABI cannot change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Change by Mark Shannon :


--
pull_requests: +24671
pull_request: https://github.com/python/cpython/pull/26021

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

Thanks everyone for the triaging and fixing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Dennis Sweeney]

Change by Dennis Sweeney :


--
pull_requests: +24662
pull_request: https://github.com/python/cpython/pull/26012

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Dennis Sweeney]

Dennis Sweeney  added the comment:

I tried some debugging code:

diff --git a/Python/ceval.c b/Python/ceval.c
index f745067069..a8668dbac2 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -4864,6 +4864,18 @@ get_exception_handler(PyCodeObject *code, int index)
 return res;
 }
 scan = skip_to_next_entry(scan);
+if (scan
+>= (unsigned char *)PyBytes_AS_STRING(code->co_exceptiontable)
++ PyBytes_GET_SIZE(code->co_exceptiontable))
+{
+printf("co_name: --\n");
+_PyObject_Dump(code->co_name);
+printf("co_filename: --\n");
+_PyObject_Dump(code->co_filename);
+printf("co_exceptiontable: -\n");
+_PyObject_Dump(code->co_exceptiontable);
+printf("\n\n\n\n\n");
+}
 }
 res.b_handler = -1;
 return res;


It output this:

Python 3.11.0a0 (heads/main-dirty:092f9ddb5e, May  9 2021, 18:45:56) [MSC 
v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from test.test_statistics import *
co_name: --
object address  : 0254B63EFB80
object refcount : 7
object type : 7FFA1C7E71C0
object type name: str
object repr : '_find_and_load'
co_filename: --
object address  : 0254B63967A0
object refcount : 76
object type : 7FFA1C7E71C0
object type name: str
object repr : ''
co_exceptiontable: -
object address  : 0254B63EB290
object refcount : 1
object type : 7FFA1C7C9A40
object type name: bytes
object repr : b'\x84\x10"\x03\xa2\x04&\x0b\xa7\x03&\x0b'


>>> unittest.main()

--
Ran 364 tests in 24.409s

OK


Here is the disassembly of the offending function:

>>> from dis import dis
>>> from importlib._bootstrap import _find_and_load
>>> dis(_find_and_load)
1024   0 LOAD_GLOBAL  0 (_ModuleLockManager)
   2 LOAD_FAST0 (name)
   4 CALL_FUNCTION1
   6 BEFORE_WITH
   8 POP_TOP

1025  10 LOAD_GLOBAL  1 (sys)
  12 LOAD_ATTR2 (modules)
  14 LOAD_METHOD  3 (get)
  16 LOAD_FAST0 (name)
  18 LOAD_GLOBAL  4 (_NEEDS_LOADING)
  20 CALL_METHOD  2
  22 STORE_FAST   2 (module)

1026  24 LOAD_FAST2 (module)
  26 LOAD_GLOBAL  4 (_NEEDS_LOADING)
  28 IS_OP0
  30 POP_JUMP_IF_FALSE   27 (to 54)

1027  32 LOAD_GLOBAL  5 (_find_and_load_unlocked)
  34 LOAD_FAST0 (name)
  36 LOAD_FAST1 (import_)
  38 CALL_FUNCTION2

1024  40 ROT_TWO
  42 LOAD_CONST   1 (None)
  44 DUP_TOP
  46 DUP_TOP
  48 CALL_FUNCTION3
  50 POP_TOP

1027  52 RETURN_VALUE

1026 >>   54 NOP

1024  56 LOAD_CONST   1 (None)
  58 DUP_TOP
  60 DUP_TOP
  62 CALL_FUNCTION3
  64 POP_TOP
  66 JUMP_FORWARD11 (to 90)
 >>   68 PUSH_EXC_INFO
  70 WITH_EXCEPT_START
  72 POP_JUMP_IF_TRUE39 (to 78)
  74 RERAISE  4
 >>   76 POP_EXCEPT_AND_RERAISE
 >>   78 POP_TOP
  80 POP_TOP
  82 POP_TOP
  84 POP_EXCEPT
  86 POP_TOP
  88 POP_TOP

1029 >>   90 LOAD_FAST2 (module)
  92 LOAD_CONST   1 (None)
  94 IS_OP0
  96 POP_JUMP_IF_FALSE   60 (to 120)

1030  98 LOAD_CONST   2 ('import of {} halted; None in 
sys.modules')

1031 100 LOAD_METHOD  6 (format)
 102 LOAD_FAST0 (name)
 104 CALL_METHOD  1

1030 106 STORE_FAST   3 (message)

1032 108 LOAD_GLOBAL  7 (ModuleNotFoundError)
 110 LOAD_FAST3 (message)
 112 LOAD_FAST0 (name)
 114 LOAD_CONST   3 (('name',))
 116 CALL_FUNCTION_KW 2
 118 

[issue40222] "Zero cost" exception handling

[Ammar Askar]

Ammar Askar  added the comment:

Seconded, also seeing the same ASAN failure on the fuzzers with a blame for 
this commit.

--
nosy: +ammar2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

To reproduce with a modern gcc:

% export ASAN_OPTIONS=detect_leaks=0:allocator_may_return_null=1:handle_segv=0
% ./configure --with-address-sanitizer --without-pymalloc
% make -j -s
% ./python -m test test_statistics
=
==51490==ERROR: AddressSanitizer: heap-buffer-overflow on address 
0x604113fd at pc 0x564ec89e0edc bp 0x7ffcba70 sp 0x7ffcba60
READ of size 1 at 0x604113fd thread T0
#0 0x564ec89e0edb in skip_to_next_entry Python/ceval.c:4798
#1 0x564ec89e0edb in get_exception_handler Python/ceval.c:4866
#2 0x564ec89e0edb in _PyEval_EvalFrameDefault Python/ceval.c:4465
#3 0x564ec8be30d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#4 0x564ec8be30d7 in _PyEval_Vector Python/ceval.c:5160
#5 0x564ec8a2b22e in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#6 0x564ec8a2b22e in object_vacall Objects/call.c:734
#7 0x564ec8a30c50 in _PyObject_CallMethodIdObjArgs Objects/call.c:825
#8 0x564ec8c5bdd7 in import_find_and_load Python/import.c:1499
#9 0x564ec8c5bdd7 in PyImport_ImportModuleLevelObject Python/import.c:1600
#10 0x564ec89ea39b in import_name Python/ceval.c:6101
#11 0x564ec89ea39b in _PyEval_EvalFrameDefault Python/ceval.c:3693
#12 0x564ec8be29ea in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#13 0x564ec8be29ea in _PyEval_Vector Python/ceval.c:5160
#14 0x564ec8be29ea in PyEval_EvalCode Python/ceval.c:1136


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

This seems to have broken the address sanitizer buildbot:

https://buildbot.python.org/all/#/builders/582/builds/116/steps/5/logs/stdio

Example error:


==28597==ERROR: AddressSanitizer: heap-buffer-overflow on address 
0x60411cbd at pc 0x55e7e3cceedc bp 0x7ffc74448490 sp 0x7ffc74448480
READ of size 1 at 0x60411cbd thread T0
#0 0x55e7e3cceedb in skip_to_next_entry Python/ceval.c:4798
#1 0x55e7e3cceedb in get_exception_handler Python/ceval.c:4866
#2 0x55e7e3cceedb in _PyEval_EvalFrameDefault Python/ceval.c:4465
#3 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#4 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#5 0x55e7e3d1922e in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#6 0x55e7e3d1922e in object_vacall Objects/call.c:734
#7 0x55e7e3d1ec50 in _PyObject_CallMethodIdObjArgs Objects/call.c:825
#8 0x55e7e3f49dd7 in import_find_and_load Python/import.c:1499
#9 0x55e7e3f49dd7 in PyImport_ImportModuleLevelObject Python/import.c:1600
#10 0x55e7e3cd839b in import_name Python/ceval.c:6101
#11 0x55e7e3cd839b in _PyEval_EvalFrameDefault Python/ceval.c:3693
#12 0x55e7e3ed09ea in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#13 0x55e7e3ed09ea in _PyEval_Vector Python/ceval.c:5160
#14 0x55e7e3ed09ea in PyEval_EvalCode Python/ceval.c:1136
#15 0x55e7e420b908 in builtin_exec_impl Python/bltinmodule.c:1065
#16 0x55e7e420b908 in builtin_exec Python/clinic/bltinmodule.c.h:371
#17 0x55e7e4196590 in cfunction_vectorcall_FASTCALL 
Objects/methodobject.c:426
#18 0x55e7e3d1a592 in PyVectorcall_Call Objects/call.c:255
#19 0x55e7e3cd15f4 in do_call_core Python/ceval.c:6028
#20 0x55e7e3cd15f4 in _PyEval_EvalFrameDefault Python/ceval.c:4283
#21 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#22 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#23 0x55e7e3cd424e in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#24 0x55e7e3cd424e in PyObject_Vectorcall Include/cpython/abstract.h:123
#25 0x55e7e3cd424e in call_function Python/ceval.c:5976
#26 0x55e7e3cd424e in _PyEval_EvalFrameDefault Python/ceval.c:4187
#27 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#28 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#29 0x55e7e3cd4384 in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#30 0x55e7e3cd4384 in PyObject_Vectorcall Include/cpython/abstract.h:123
#31 0x55e7e3cd4384 in call_function Python/ceval.c:5976
#32 0x55e7e3cd4384 in _PyEval_EvalFrameDefault Python/ceval.c:4204
#33 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#34 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#35 0x55e7e3ce0934 in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#36 0x55e7e3ce0934 in PyObject_Vectorcall Include/cpython/abstract.h:123
#37 0x55e7e3ce0934 in call_function Python/ceval.c:5976
#38 0x55e7e3ce0934 in _PyEval_EvalFrameDefault Python/ceval.c:4219
#39 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#40 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#41 0x55e7e3ce0934 in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#42 0x55e7e3ce0934 in PyObject_Vectorcall Include/cpython/abstract.h:123
#43 0x55e7e3ce0934 in call_function Python/ceval.c:5976
#44 0x55e7e3ce0934 in _PyEval_EvalFrameDefault Python/ceval.c:4219
#45 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#46 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#47 0x55e7e3ce0934 in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#48 0x55e7e3ce0934 in PyObject_Vectorcall Include/cpython/abstract.h:123
#49 0x55e7e3ce0934 in call_function Python/ceval.c:5976
#50 0x55e7e3ce0934 in _PyEval_EvalFrameDefault Python/ceval.c:4219
#51 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#52 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#53 0x55e7e3cd424e in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#54 0x55e7e3cd424e in PyObject_Vectorcall Include/cpython/abstract.h:123
#55 0x55e7e3cd424e in call_function Python/ceval.c:5976
#56 0x55e7e3cd424e in _PyEval_EvalFrameDefault Python/ceval.c:4187
#57 0x55e7e3ed10d7 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:46
#58 0x55e7e3ed10d7 in _PyEval_Vector Python/ceval.c:5160
#59 0x55e7e3cd424e in _PyObject_VectorcallTstate 
Include/cpython/abstract.h:114
#60 0x55e7e3cd424e in PyObject_Vectorcall Include/cpython/abstract.h:123
#61 0x55e7e3cd424e in call_function Python/ceval.c:5976
#62 0x55e7e3cd424e in _PyEval_EvalFrameDefault Python/ceval.c:4187
#63 0x55e7e3ed10d7 in _PyEval_EvalFrame 

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Change by Mark Shannon :


--
keywords: +patch
pull_requests: +24440
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25729

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

I've played around with a few formats, and what I've ended up with is this:

1. Use the >> marker for for exception targets, as well as normal branch 
targets.
2. Add a text version of the exception handler table at the end of the 
disassembly.

This has all the information, without too much visual clutter.

The function `f` above looks like this:
>>> dis.dis(f)
  2   0 NOP

  3   2 LOAD_CONST   1 (1)
  4 LOAD_CONST   2 (0)
  6 BINARY_TRUE_DIVIDE
  8 POP_TOP
 10 NOP
 12 LOAD_CONST   0 (None)
 14 RETURN_VALUE
>>   16 NOP
 18 PUSH_EXC_INFO

  4  20 POP_TOP
 22 POP_TOP
 24 POP_TOP

  5  26 NOP
 28 POP_EXCEPT
 30 LOAD_CONST   3 ('fail')
 32 RETURN_VALUE
>>   34 POP_EXCEPT_AND_RERAISE
ExceptionTable:
  2 to 8 -> 16 (depth 0)
  18 to 24 -> 34 (depth 3) lasti


The 'lasti' field indicates that the offset of the last instruction is pushed 
to the stack, which is needed for cleanup-then-reraise code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

Responding to Serhiy's suggestions:
1 Add another column:
   Adding another column makes for lots of repetition in larger try blocks, and 
pushes useful information further to the right.
2 Add pseudo-instructions
  I find those misleading, as they aren't really there, and probably won't even 
correspond to the original SETUP_XXX instructions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

> BTW, what are the three POP_TOP op codes in a row popping?

When exceptions are pushed to the stack, they are pushed as a triple: (exc, 
type, traceback)
so when we pop them, we need three pops.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
nosy: +erlendaasland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Guido van Rossum  added the comment:

I like Serhiy’s idea.

BTW, what are the three POP_TOP op codes in a row popping?

--
nosy: +Guido.van.Rossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

We can add a new column for the offset or the index of the error handler. Or 
add pseudo-instructions (which do not correspond to any bytecode) at boundaries 
of the code with some error handler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

Mark Shannon  added the comment:

The changes to pyc format aren't user visible so shouldn't matter,
but what about the dis output?

Consider this program:

def f():
try:
1/0
except:
return "fail"

Currently it compiles to:
  2   0 SETUP_FINALLY7 (to 16)

  3   2 LOAD_CONST   1 (1)
  4 LOAD_CONST   2 (0)
  6 BINARY_TRUE_DIVIDE
  8 POP_TOP
 10 POP_BLOCK
 12 LOAD_CONST   0 (None)
 14 RETURN_VALUE

  4 >>   16 POP_TOP
 18 POP_TOP
 20 POP_TOP

  5  22 POP_EXCEPT
 24 LOAD_CONST   3 ('fail')
 26 RETURN_VALUE

With zero-cost exception handling, it will compile to something like:  
  2   0 NOP
  3   2 LOAD_CONST   1 (1)
  4 LOAD_CONST   2 (0)
  6 BINARY_TRUE_DIVIDE
  8 POP_TOP
 10 LOAD_CONST   0 (None)
 12 RETURN_VALUE

  None   14 PUSH_EXCEPT

  4  16 POP_TOP
 18 POP_TOP
 20 POP_TOP

  5  22 POP_EXCEPT
 24 LOAD_CONST   3 ('fail')
 26 RETURN_VALUE

(There are additional optimizations that should be applied, but those are a 
separate issue)

The problem is that the exception handling flow is no longer visible.
Should we add it back in somehow, or just append the exception jump table?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

The only observable changes will be changes in the code object: new attributes 
and constructor parameters, changed .pyc format, dis output, etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Chris Jerdonek]

Chris Jerdonek  added the comment:

To clarify, would there be any observable difference in behavior aside from 
speed? And would there be any limitations in when the speedup can be applied?

--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Dong-hee Na]

Dong-hee Na  added the comment:

+1

--
nosy: +corona10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Yonatan Goldschmidt]

Change by Yonatan Goldschmidt :


--
nosy: +Yonatan Goldschmidt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Raymond Hettinger]

Raymond Hettinger  added the comment:

This is an exciting prospect.  Am looking forward to it :-)

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Shantanu]

Change by Shantanu :


--
nosy: +hauntsaninja

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

+1! I was going to implement this, but first I wanted to implement support of 
line number ranges instead of just line numbers (co_lineno). We need to design 
some compact portable format for address to address mapping (or address range 
to address mapping if it is more efficient).

Are you already working on this Mark? I would be glad to make a review.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40222] "Zero cost" exception handling

[Mark Shannon]

New submission from Mark Shannon :

C++ and Java support what is known as "zero cost" exception handling.
The "zero cost" refers to the cost when no exception is raised. There is still 
a cost when exceptions are thrown.

The basic principle is that the compiler generates tables indicating where 
control should be transferred to when an exception is raised. When no exception 
is raised, there is no runtime overhead.

(C)Python should support "zero cost" exceptions.


Now that the bytecodes for exception handling are regular (meaning that their 
stack effect can be statically determined) it is possible for the bytecode 
compiler to emit exception handling tables.

Doing so would have two main benefits.
1. "try" and "with" statements would be faster (and "async for", but that is an 
implementation detail).
2. Calls to Python functions would be faster as frame objects would be 
considerably smaller. Currently each frame carries 240 bytes of overhead for 
exception handling.

--
assignee: Mark.Shannon
messages: 365974
nosy: Mark.Shannon
priority: normal
severity: normal
status: open
title: "Zero cost" exception handling
type: performance

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com