[Python-Dev] PEP 490: Chain exceptions at C level
Hi,
I didn't get much feedback on this PEP. Since the Python 3.6 branch is
open (default), it's probably better to push such change in the
beginning of the 3.6 cycle, to catch issues earlier.
Are you ok to chain exceptions at C level by default?
Relatedi issue: https://bugs.python.org/issue23763
Victor
PEP: 490
Title: Chain exceptions at C level
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 25-March-2015
Python-Version: 3.6
Abstract
Chain exceptions at C level, as already done at Python level.
Rationale
=
Python 3 introduced a new killer feature: exceptions are chained by
default, PEP 3134.
Example::
try:
raise TypeError("err1")
except TypeError:
raise ValueError("err2")
Output::
Traceback (most recent call last):
File "test.py", line 2, in
raise TypeError("err1")
TypeError: err1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 4, in
raise ValueError("err2")
ValueError: err2
Exceptions are chained by default in Python code, but not in
extensions written in C.
A new private ``_PyErr_ChainExceptions()`` function was introduced in
Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
explicitly to chain exceptions and its usage is not trivial.
Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
module to chain the previous ``OSError`` to a new ``ZipImportError``
exception::
PyObject *exc, *val, *tb;
PyErr_Fetch(&exc, &val, &tb);
PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
_PyErr_ChainExceptions(exc, val, tb);
This PEP proposes to also chain exceptions automatically at C level to
stay consistent and give more information on failures to help
debugging. The previous example becomes simply::
PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
Proposal
Modify PyErr_*() functions to chain exceptions
--
Modify C functions raising exceptions of the Python C API to
automatically chain exceptions: modify ``PyErr_SetString()``,
``PyErr_Format()``, ``PyErr_SetNone()``, etc.
Modify functions to not chain exceptions
Keeping the previous exception is not always interesting when the new
exception contains information of the previous exception or even more
information, especially when the two exceptions have the same type.
Example of an useless exception chain with ``int(str)``::
TypeError: a bytes-like object is required, not 'type'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
TypeError: int() argument must be a string, a bytes-like object or
a number, not 'type'
The new ``TypeError`` exception contains more information than the
previous exception. The previous exception should be hidden.
The ``PyErr_Clear()`` function can be called to clear the current
exception before raising a new exception, to not chain the current
exception with a new exception.
Modify functions to chain exceptions
Some functions save and then restore the current exception. If a new
exception is raised, the exception is currently displayed into
sys.stderr or ignored depending on the function. Some of these
functions should be modified to chain exceptions instead.
Examples of function ignoring the new exception(s):
* ``ptrace_enter_call()``: ignore exception
* ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
* ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
by trying to display the bootstrap function to ``sys.stderr``
* ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
exception raised by looking for a key in the dictionary
* ``_PyErr_TrySetFromCause()``: ignore exception
* ``PyFrame_LocalsToFast()``: ignore exception raised by
``dict_to_map()``
* ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
to debug, to inspect a running process, it should not modify the
Python state.
* ``Py_ReprLeave()``: ignore exception "because there is no way to
report them"
* ``type_dealloc()``: ignore exception raised by
``remove_all_subclasses()``
* ``PyObject_ClearWeakRefs()``: ignore exception?
* ``call_exc_trace()``, ``call_trace_protected()``: ignore exception
* ``remove_importlib_frames()``: ignore exception
* ``do_mktuple()``, helper used by ``Py_BuildValue()`` for example:
ignore exception?
* ``flush_io()``: ignore exception
* ``sys_write()``, ``sys_format()``: ignore exception
* ``_PyTraceback_Add()``: ignore exception
* ``PyTraceBack_Print()``: ignore exception
Examples of function displaying the new exception to ``sys.stderr``:
* ``atexit_callfuncs()``: display exceptio
[Python-Dev] Unbound locals in class scopes
Hello, There appeared a question in the discussion on http://bugs.python.org/issue24129 about documenting the behavior that unbound local variables in a class definition do not follow the normal rules. Guido said 13 years ago that this behavior should not be changed: https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 490: Chain exceptions at C level
Victor Stinner schrieb am 20.06.2015 um 09:30: > Are you ok to chain exceptions at C level by default? I agree that it can be a bit non-obvious where exceptions are chained and when they are not and my guess is that most C code simply doesn't take care of chaining exceptions at all. If only because it was ported from Python 2 to the point that it worked in Python 3. Almost no test suite will include dedicated tests for chained exception handling in C code, so flaws in this area would rarely be noticed. I'm rather for this proposal (say, +0.3) for consistency reasons, but it will break code that was written with the assumption that exception chaining doesn't happen for the current exception automatically, and also code that incidentally gets it right. I don't think I have an idea of what's more common in C code - that exception chaining is wanted, or that the current exception is intentionally meant to be replaced by another. And that is the crux in your proposal: you're changing the default behaviour into its opposite. In order to do that, it should be reasonably likely that the current standard behaviour is not intended in more than half of the cases. I find that difficult to estimate. Stefan ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async __setitem__
On 19 June 2015 at 22:56, Martin Teichmann wrote: > to get something out of the database. But getting something in, I have > to write something like > > await table.set(key, value) > > It would be cool if I could just write > > await table[key] = value You've introduced an ambiguity here, though, as you're applying async/await to a statement without a leading keyword. In "await expr" it's clear "expr" is expected to produce an Awaitable, and the await expression waits for it. In "async for" and "async with" it's clearly a modifier on the respective keyword, and hence serves as a good mnemonic for switching to the asynchronous variants of the relevant protocols. But what does an "asynchronous assignment" do? Do we need __asetattr__ and __asetitem__ protocols, and only allow it when the target is a subscript operation or an attribute? What if we're assigning to multiple targets, do the run in parallel? How is tuple unpacking handled? How is augmented assignment handled? If we allow asynchronous assignment, do we allow asynchronous deletion as well? As you start working through some of those possible implications of offering an asynchronous assignment syntax, the explicit method based "await table.set(key, value)" construct may not look so bad after all :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 490: Chain exceptions at C level
On 20 June 2015 at 19:00, Stefan Behnel wrote: > And that is the crux in your proposal: you're changing the default > behaviour into its opposite. In order to do that, it should be reasonably > likely that the current standard behaviour is not intended in more than > half of the cases. I find that difficult to estimate. I think in this case the design decision can be made more on the basis of how difficult it is to flip the behaviour to the non-default case: Status quo: "Dear CPython core developers, why do you hate us so? Signed, everyone that has ever needed to manually chain exceptions at the C level using the current C API" (OK, it's not *that* bad as C level coding goes, but it's still pretty arcane, requiring a lot of knowledge of exception handling implementation details) With Victor's PEP: "Call PyErr_Clear() before setting your exception to disable the implicit exception chaining in 3.6+ in a way that remains entirely compatible with older Python versions" Given that significant difference in relative usability, +1 from me for changing the default behaviour. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbound locals in class scopes
On 06/20/2015 02:51 AM, Ivan Levkivskyi wrote:
Hello,
There appeared a question in the discussion on
http://bugs.python.org/issue24129 about documenting the behavior that
unbound local variables in a class definition do not follow the normal rules.
Guido said 13 years ago that this behavior should not be changed:
https://mail.python.org/pipermail/python-dev/2002-April/023428.html,
however, things changed a bit in Python 3.4 with the introduction of the
LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a
desired/expected behavior.
Guido's comment still stands as far as references inside methods work in
regards to the class body. (they must use a self name to access the class
name space.) But the execution of the class body does use lexical scope,
otherwise it would print xtop instead of xlocal here.
x = "xtop"
y = "ytop"
def func():
x = "xlocal"
y = "ylocal"
class C:
print(x)
print(y)
y = 1
func()
prints
xlocal
ytop
Maybe a better way to put this is, should the above be the same as this?
>>> x = "xtop"
>>> y = "ytop"
>>> def func():
... x = "xlocal"
... y = "ylocal"
... def _C():
... print(x)
... print(y)
... y = 1
... return locals()
... C = type("C", (), _C())
...
>>> func()
xlocal
Traceback (most recent call last):
File "", line 1, in
File "", line 9, in func
File "", line 6, in _C
UnboundLocalError: local variable 'y' referenced before assignment
I think yes, but I'm not sure how it may be different in other ways.
Cheers,
Ron
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unbound locals in class scopes
On 06/20/2015 12:12 PM, Ron Adam wrote: On 06/20/2015 02:51 AM, Ivan Levkivskyi wrote: Guido said 13 years ago that this behavior should not be changed: https://mail.python.org/pipermail/python-dev/2002-April/023428.html, however, things changed a bit in Python 3.4 with the introduction of the LOAD_CLASSDEREF opcode. I just wanted to double-check whether it is still a desired/expected behavior. Guido's comment still stands as far as references inside methods work in regards to the class body. (they must use a self name to access the class name space.) But the execution of the class body does use lexical scope, otherwise it would print xtop instead of xlocal here. Minor corrections: Methods can access but not write to the class scope without using self. So that is also equivalent to the function version using type(). The methods capture the closure they were defined in, which is interesting. And the self name refers to the object's names space not the class name space. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async __setitem__
Hi Martin, Actually, I think that it's better to adopt a bit different design: async with db.transaction(): table[key1] = 'a' table[key2] = 'b' And then, in __aexit__ you should flush the updates to the database in bulk. Usually, when working with an ORM, you need to update more than one field anyways. And if it's just one, 'await table.set(key, val)' looks more obvious than 'await table[key] = val'. I'd also recommend you to prohibit __setitem__ and __setattr__ outside of a transaction. Yury On 2015-06-19 8:56 AM, Martin Teichmann wrote: Hi everyone, I was really happy to see PEP 492 - that is really a big step forward. It makes many programming constructs possible in combination with asyncio, which have been cumbersome or impossible until now. One thing that still is impossible is to have __setitem__ be used asynchronously. As an example, I am working with a database that I access over the network, and it is easy to write data = await table[key] to get something out of the database. But getting something in, I have to write something like await table.set(key, value) It would be cool if I could just write await table[key] = value which would, I think, fit in nicely with all of PEP 492. That said, I was hesitating which keyword to use, await or async. PEP 492 says something like async is an adjective, await a command. Then for setitem, technically, I would need an adverb, so "asyncly table[key] = value". But I didn't want to introduce yet another keyword (two different ones for the same thing are already confusing enough), so I arbitrarily chose await. If we are really going for it (I don't see the use case yet, I just mention it for completeness), we could also be extend this to other points where variables are set, e.g. "for await i in something():" which is a bit similar to "async for i in something()" yet different. Greetings Martin ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 490: Chain exceptions at C level
On 20.06.2015 09:30, Victor Stinner wrote:
> Hi,
>
> I didn't get much feedback on this PEP. Since the Python 3.6 branch is
> open (default), it's probably better to push such change in the
> beginning of the 3.6 cycle, to catch issues earlier.
>
> Are you ok to chain exceptions at C level by default?
I think it's a good idea to make C APIs available to
simplify chaining exceptions at the C level, but don't
believe that always doing this by default is a good idea.
It should really be a case-by-case decision, IMO.
Note that Python exceptions are cheap to raise in C
(very much unlike in Python), so making this more
expensive by default would introduce a significant
overhead - without much proven benefit.
More below...
> PEP: 490
> Title: Chain exceptions at C level
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 25-March-2015
> Python-Version: 3.6
>
>
> Abstract
>
>
> Chain exceptions at C level, as already done at Python level.
>
>
> Rationale
> =
>
> Python 3 introduced a new killer feature: exceptions are chained by
> default, PEP 3134.
>
> Example::
>
> try:
> raise TypeError("err1")
> except TypeError:
> raise ValueError("err2")
>
> Output::
>
> Traceback (most recent call last):
> File "test.py", line 2, in
> raise TypeError("err1")
> TypeError: err1
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "test.py", line 4, in
> raise ValueError("err2")
> ValueError: err2
>
> Exceptions are chained by default in Python code, but not in
> extensions written in C.
>
> A new private ``_PyErr_ChainExceptions()`` function was introduced in
> Python 3.4.3 and 3.5 to chain exceptions. Currently, it must be called
> explicitly to chain exceptions and its usage is not trivial.
>
> Example of ``_PyErr_ChainExceptions()`` usage from the ``zipimport``
> module to chain the previous ``OSError`` to a new ``ZipImportError``
> exception::
>
> PyObject *exc, *val, *tb;
> PyErr_Fetch(&exc, &val, &tb);
> PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
> _PyErr_ChainExceptions(exc, val, tb);
>
> This PEP proposes to also chain exceptions automatically at C level to
> stay consistent and give more information on failures to help
> debugging. The previous example becomes simply::
>
> PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
>
>
> Proposal
>
>
> Modify PyErr_*() functions to chain exceptions
> --
>
> Modify C functions raising exceptions of the Python C API to
> automatically chain exceptions: modify ``PyErr_SetString()``,
> ``PyErr_Format()``, ``PyErr_SetNone()``, etc.
>
>
> Modify functions to not chain exceptions
>
>
> Keeping the previous exception is not always interesting when the new
> exception contains information of the previous exception or even more
> information, especially when the two exceptions have the same type.
>
> Example of an useless exception chain with ``int(str)``::
>
> TypeError: a bytes-like object is required, not 'type'
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "", line 1, in
> TypeError: int() argument must be a string, a bytes-like object or
> a number, not 'type'
>
> The new ``TypeError`` exception contains more information than the
> previous exception. The previous exception should be hidden.
>
> The ``PyErr_Clear()`` function can be called to clear the current
> exception before raising a new exception, to not chain the current
> exception with a new exception.
>
>
> Modify functions to chain exceptions
>
>
> Some functions save and then restore the current exception. If a new
> exception is raised, the exception is currently displayed into
> sys.stderr or ignored depending on the function. Some of these
> functions should be modified to chain exceptions instead.
>
> Examples of function ignoring the new exception(s):
>
> * ``ptrace_enter_call()``: ignore exception
> * ``subprocess_fork_exec()``: ignore exception raised by enable_gc()
> * ``t_bootstrap()`` of the ``_thread`` module: ignore exception raised
> by trying to display the bootstrap function to ``sys.stderr``
> * ``PyDict_GetItem()``, ``_PyDict_GetItem_KnownHash()``: ignore
> exception raised by looking for a key in the dictionary
> * ``_PyErr_TrySetFromCause()``: ignore exception
> * ``PyFrame_LocalsToFast()``: ignore exception raised by
> ``dict_to_map()``
> * ``_PyObject_Dump()``: ignore exception. ``_PyObject_Dump()`` is used
> to debug, to inspect a running process, it should not modify the
> Python state.
> * ``Py_ReprLeave()``: ignore
Re: [Python-Dev] PEP 490: Chain exceptions at C level
On 2015-06-20 3:16 PM, M.-A. Lemburg wrote: On 20.06.2015 09:30, Victor Stinner wrote: >Hi, > >I didn't get much feedback on this PEP. Since the Python 3.6 branch is >open (default), it's probably better to push such change in the >beginning of the 3.6 cycle, to catch issues earlier. > >Are you ok to chain exceptions at C level by default? I think it's a good idea to make C APIs available to simplify chaining exceptions at the C level, but don't believe that always doing this by default is a good idea. It should really be a case-by-case decision, IMO. Exactly my thoughts. Maybe PyErr_SetStringChained, PyErr_FormatChained, etc is better? Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async __setitem__
Nick Coghlan wrote: What if we're assigning to multiple targets, do the run in parallel? How is tuple unpacking handled? How is augmented assignment handled? If we allow asynchronous assignment, do we allow asynchronous deletion as well? Yeah, we'd kind of be letting the camel's nose in here. There's no obvious place to stop until *every* dunder method has an async counterpart. -- Greg ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
