[Python-Dev] Re: Python 3.11 bytecode and exception table
Hi Matthieu, Yes I am interested. Please ping me for PR reviews or any progress updates. Thanks Irit > On 5 Jul 2022, at 20:27, Matthieu Dartiailh wrote: > > Hi Irit, hi Patrick, > > Thanks for your quick answers. > > First thanks Patrick, it seems I went back to the stable docs at one point > without noticing it and hence I missed the new opcodes. > > Thanks Irit for the clarification regarding the pseudo-instructions use in > dis. > > Regarding the existence of nested try/except I believe a we could have 2 > SETUP_* followed by 2 POP_BLOCK so I am not sure what issue you see there. > However if we can have exception tables with two rows such as (1, 3, ...) and > (2, 4, ...) then yes I will have an issue. I guess I will have to try > implementing something and try to roundtrip on as many examples as possible. > Would you be interested in being posted about my progress ? > > Best > > Matthieu ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B2G4LQ3WPAYEUEZX6LURWO336EMZTPC2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python 3.11 bytecode and exception table
Hi Matthieu, The dis output for this function in 3.12 is the same as it is in 3.11. The pseudo-instructions are emitted by the compiler's codegen stage, but never make it to compiled bytecode. They are removed or replaced by real opcodes before the code object is created. The recent change to the dis module that you mentioned did not change how the disassembly of bytecode gets displayed. Rather, it added the pseudo-instructions to the opcodes list so that we have access to their mnemonics from python. This is a step towards exposing intermediate compilation steps to python (for unit tests, etc). BTW - part of this will require writing some test utilities for cpython that let us specify and compare opcode sequences, similar to what you have in bytecode. As for deconstructing the exception table and planting the pseudo instructions back into the code - it would be nice if dis could do that, but we may need to settle for an approximation because I'm not sure the exact block structure can be reliably reconstructed from the exception table at the moment. I may be wrong. Having a SETUP_*/POP_BLOCK for each line in the exception table is not going to be correct - there can be nested try-except blocks, for instance, and even without them the compiler can emit the code of an except block in non-contiguous order (in https://github.com/python/cpython/pull/93622 I fixed one of those cases to reduce the size of the exception table, but it wasn't a correctness bug). Irit On Tue, Jul 5, 2022 at 9:27 AM Matthieu Dartiailh wrote: > Hi all, > > I am the current maintainer of bytecode ( > https://github.com/MatthieuDartiailh/bytecode) which is a library to > perform assembly and disassembly of Python bytecode. The library was > created by V. Stinner. > > I started looking in Python 3.11 support in bytecode, I read > Objects/exception_handling_notes.txt and I have a couple of questions > regarding the exception table: > > Currently bytecode exposes three level of abstractions: > - the concrete level in which one deals with instruction offset for > jumps and explicit indexing into the known constants and names > - the bytecode level which uses labels for jumps and allow non integer > argument to instructions > - the cfg level which provides basic blocks delineation over the > bytecode level > > So my first idea was to directly expose the unpacked exception table > (start, stop, target, stack_depth, last_i) at the concrete level and use > pseudo-instruction and labels at the bytecode level. At this point of my > reflections, I saw > https://github.com/python/cpython/commit/c57aad777afc6c0b382981ee9e4bc94c03bf5f68 > about adding pseudo-instructionto dis output in 3.12 and though it would > line up quite nicely. Reading through, I got curious about how SETUP_WITH > handled popping one extra item from the stack so I went to look at dis > results on a couple of small examples. I tried on 3.10 and 3.11b3 (for some > reasons I cannot compile main at a391b74d on windows). > > I looked at simple things and got a bit surprised: > > Disassembling: > def f(): > try: > a = 1 > except: > raise > > I get on 3.11: > 1 0 RESUME 0 > > 2 2 NOP > > 3 4 LOAD_CONST 1 (1) > 6 STORE_FAST 0 (a) > 8 LOAD_CONST 0 (None) > 10 RETURN_VALUE > >> 12 PUSH_EXC_INFO > > 4 14 POP_TOP > > 5 16 RAISE_VARARGS0 > >> 18 COPY 3 > 20 POP_EXCEPT > 22 RERAISE 1 > ExceptionTable: > 4 to 6 -> 12 [0] > 12 to 16 -> 18 [1] lasti > > On 3.10: > 2 0 SETUP_FINALLY5 (to 12) > > 3 2 LOAD_CONST 1 (1) > 4 STORE_FAST 0 (a) > 6 POP_BLOCK > 8 LOAD_CONST 0 (None) > 10 RETURN_VALUE > > 4 >> 12 POP_TOP > 14 POP_TOP > 16 POP_TOP > > 5 18 RAISE_VARARGS0 > > This surprised me on two levels: > - first I have never seen the RESUME opcode and it is currently not > documented > - my second surprise comes from the second entry in the exception table. > At first I failed to see why it was needed but writing this I realize it > corresponds to the explicit handling of exception propagation to the > caller. Since I cannot compile 3.12 ATM I am wondering how this plays with > pseudo-instruction: in particular are pseudo-instructions generated for all > entries in the exception table ? > > My initial idea was to have a SETUP_FINALLY/SETUP_CLEANUP - POP_BLOCK pair > for each line in the exception table and label for the jump target. But I > realize it means we will have many such pairs than in 3.10. It is fine by > me but I wondered what choice was made in 3.12 dis and if this approach > made sense. > > Best regards > >
[Python-Dev] Re: import * and __future__ imports
On Mon, Mar 28, 2022 at 4:44 PM Guido van Rossum wrote: > > "Future" imports are special to the parser, and they may also set a flag > for the runtime to alter its behavior, but they are intentionally not > treated specially by code generation, so they are still properly imported. > However the presence of the imported thing is not used by the runtime to > determine its behavior; those flags are stored elsewhere guided by the code > generator. > Right, this is why it's confusing when the object is there for no reason (the future import did not have any impact on the module, but the object showed up via an import *). > I don't think there's anything to do here. > How about the suggestion in https://bugs.python.org/issue26120 ? It's about these objects showing up in pydoc (both when the __future__ had an impact and when it didn't - in either case it's not an interesting part of the module's API). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JVHTGPOZX72J73SUVKGEWBXHKK4KIFAV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] import * and __future__ imports
If you have a __future__ import in a script, and you import * from it in another script, the object for this future appears in the dir() of the other script, even though the __future__ import has no effect there. % cat x.py from __future__ import annotations % cat y.py from x import * print(dir()) class D: def f(self, a: D): return 42 % ./python.exe y.py ['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'annotations'] Traceback (most recent call last): File "/Users/iritkatriel/src/cpython-654/y.py", line 5, in class D: File "/Users/iritkatriel/src/cpython-654/y.py", line 6, in D def f(self, a: D): ^ NameError: name 'D' is not defined I think we should change import * to exclude the __future__ import objects, and perhaps also to not show them in dir(x). Any objections? This came up in the discussion about https://bugs.python.org/issue26120 . See the attached PR for a technique we can use to identify those objects. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3BUUCY2OHZILBJLOTS6I33M5YW4INDW3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
Stefan, There two separate issues here. One is the timing of committing changes into cython, and the other is the process by which the cython devs learn about cpython development. On the first issue, you wrote: I'm reluctant to working on adapting Cython during alphas, because it > happened more than once that incompatible changes in CPython were rolled > back or modified again during alpha, beta and rc phases. That means more > work for me and the Cython project, and its users. Code that Cython users > generate and release on their side with a release version of Cython will > then be broken, and sometimes even more broken than with an older Cython > release. > I saw in your patch that you make changes such that they impact only the new cpython version. So for old versions the generated code should not be broken. Surely you don't guarantee that cython code generated for an alpha version of cpython will work on later versions as well? Users who generate code for an alpha version should regenerate it for the next alpha and for beta, right? On the second issue: I don't have the capacity to follow all relevant changes in CPython, > incompatible or not. We get that, and this is why we're asking to work with you on cython updates so that this will be easier for all of us. There are a number of cpython core devs who would like to help cython maintenance. We realise how important and thinly resourced cython is, and we want to reduce your maintenance burden. With better communication we could find ways to do that. Returning to the issue that started this thread - how do you suggest we proceed with the exc_info change? Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MBGJFYWKBFBUKNA2MXJ5THX5M657J5WH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
Miro, I have offered before and my offer still stands to help fix this. This was already fixed in the cython main branch by Stefan. The discussion now is about when to backport it to cython 0.29. I'm actually working on the backport now (learning cython in the process). But we will need to come up with a release plan that doesn't make me revert the cpython changes until after the 3.11 beta is released, because that would mean that I can only make them in 3.12. On Tue, Feb 1, 2022 at 6:53 PM Miro Hrončok wrote: > On 01. 02. 22 17:42, Victor Stinner wrote: > > The problem right now is the pressure put on Cython maintainers to fix > > Cython as soon as possible. IMO core developers who introduce > > incompatible changes should be more involved in the Cython changes, > > since Cython is a **key component** of the Python ecosystem. IMO > > knowing that a change breaks Cython and relying on "the community" to > > fix it is not a nice move. Well, that's my opinion;-) > > As the Fedora Python maintainer, I agree with this opinion. Broken Cython > means > we cannot actually test the next pre-release of CPython until it is fixed. > And > the CPython contributors who introduced the chnage are the most equipped > ones > to help fix it. > > I understand the desire to innovate fast, but making sure Cython works > should > be an essential part of the innovation process (even while Cython is not > part > of the CPython source tree, it's part of the bigger picture). > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/K7LZAJTGDBFDM5TEQE7EALZMXQTCMQUS/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L564CUXVMFZ4YHDNPM6K47FRBGUT5FDH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Please update Cython *before* introcuding C API incompatible changes in Python
_PyErr_StackItem is not part of the C API, it's an internal struct that cython accesses directly. On Tue, Feb 1, 2022 at 3:42 PM Christian Heimes wrote: > On 01/02/2022 16.08, Victor Stinner wrote: > > -- > > > > I would prefer to introduce C API incompatible changes differently: > > first fix Cython, and *then* introduce the change. > > > > - (1) Propose a Cython PR and get it merged > > - (2) Wait until a new Cython version is released > > - (3) If possible, wait until numpy is released with regenerated Cython > code > > - (4) Introduce the incompatible change in Python > > > > Note: Fedora doesn't need (3) since we always regenerated Cython code in > numpy. > > Hi, > > this is a reasonable request for beta releases, but IMHO it is not > feasible for alphas. During alphas we want to innovate fast and play > around. Your proposal would slow down innovation and impose additional > burden on core developers. > > There are more code binding generators than just Cython. Shouldn't we > work with cffi, SWIG, sip, pybind11, and PyO3 developers as well? I care > for cffi and PyO3, too... > > I would prefer if we can get Cython and all the other code generator and > bindings library off the unstable C-API. They should use the limited API > instead. If they require any C-APIs outside the limited API, then we > should investigate and figure something out. > > Christian > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/7UD4FZC7ANLR646CNP4HJ2WNWLFRYL7I/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KFFTVAJI4QVKNCDIIVU5HEHISQJI5ZWI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Increase of Spammy PRs and PR reviews
A non-core approval changes the label from "awaiting review" to "awaiting core review". I find this useful, and occasionally filter on "awaiting core review" because it indicates that at least one other person found the PR sound / interesting. I would not like this to change - I think high quality reviews from non-core contributors are valuable for the project and are a very quick way for the contributor to get the right kind of attention from core devs. If people spam the approvals (i.e., approve PRs without reviewing them) then the distinction between the labels becomes meaningless, of course. Though I do wonder what the motivation for doing that repeatedly would be. My basic assumption is that people usually try not to make fools of themselves. Note that contributors can still approve a perfect PR - a quality review in this case would include a brief comment indicating that you understand the change and perhaps why you find it valuable. On the matter of spammy PRs - I agree with both Rupert and Ethan (F): trivial PRs can add value, and a large number of them can be annoying. You can add value and be annoying at the same time (my triage work on b.p.o is probably in this category. Thankfully it's clear I'm not after "triage points", because there aren't any). At the end of the day, it doesn't really matter what a contributor's motivation is - it's up to the core devs to decide which PRs are valuable enough to merge, or even to review, and who they enjoy working with. These things tend to sort themselves out on their own. I don't think we need to restrict access for non-core contributors compared to the status quo. What I do think we need is to make it easier for core devs to close issues and PRs. We have a huge pile of open issues and PRs, some of which we know will never happen (stale or otherwise) and we don't close them because it's an unpleasant task to let someone down, and sometimes they argue, and we're volunteers and why should we bother with this emotional labour. Through triage I found many 6 year old issues that, once I refreshed and perhaps put an 'easy' label on, got fixed. The useless issues we don't want to close are obscuring the ones we can and should fix. I'm sure this has been discussed before. My only idea is that we formalize some guidelines/criteria for closing old issues and PRs that make it more of a joint decision of the core devs and less of a personal issue between the core dev and the contributor. I would not suggest a blanket 'close issues/PRs with no activity for X months', as I said I found useful 6 year old issues. It has to be more sophisticated than that. In summary - my view is that rather than making contributors contribute less, we should be more effective in reviewing the contributions, including rejecting those that should be rejected. On Sun, Jan 30, 2022 at 8:06 AM Ethan Smith wrote: > > > On Sat, Jan 29, 2022 at 7:38 PM Inada Naoki > wrote: > >> On Sun, Jan 30, 2022 at 12:03 PM Ethan Smith wrote: >> > >> > As a non-committer, I want to make a plea for non-committer approval >> reviews, or at least that they have a place. When asked how outsiders can >> contribute I frequently see "review open PRs" as a suggested way of >> contributing to CPython. Not being able to approve PRs that are good would >> be a barrier to those contributions. >> > >> > Furthermore, I am collaborating with a couple of core devs, it would >> make collaboration more difficult if I couldn't review their work and say >> that I thought the changes looked good. >> > >> >> You can still write a review comment, including "LGTM". > > > Fair enough, I suppose commenting with "LGTM" works just as well. > > >> What you can >> not is labeling PR as "Approved." >> So I don't think it makes collaboration difficult. >> > By preventing approval from others, we can easily find PRs approved >> from core-devs or triage members but not merged yet. >> > >> > I know "drive by approvals" are annoying but I think it is >> unfortunately part of open source projects. >> > >> >> Sorry, but I don't think so. >> > > Well, if you disallow drive-by approvals, you will still get drive-by LGTM > comments. People seem to believe that adding their approval will expedite a > PR merge, for some reason (or want to bump a stale PR in hopes of a quicker > merge). > > >> >> -- >> Inada Naoki >> > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/QJ5QBB4XCFU3DFSOFBRUJB5XC4UMX7TK/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/
[Python-Dev] Re: PEP 654 except* formatting
We can drop except. Say:
try:
..
handle T1:
…
handle T2:
…
Or ‘catch’, or something else.
> On 3 Oct 2021, at 19:12, MRAB wrote:
>
> On 2021-10-03 18:50, Brandt Bucher wrote:
>> Łukasz Langa wrote:
>>> My idea is this:
>>> try:
>>>...
>>> except group E as e:
>>>...
>>> except group E1, T2 as e:
>>>...
>>> Should be doable given the magical match-case contextual keywords
>>> precedent. This looks nice and is explicit, since you will always get an
>>> ExceptionGroup instance under `e`.
>> Heh, we crossed posts with the soft keywords. I like your idea (“except
>> group”) better than mine (“except each”).
>> If we want to use an existing keyword instead of a soft keyword, how
> about "except in E as e:".
>
> The disadvantage, as I see it, from a linguistic point of view, is that
> "except in" could be read as "excluding", but, then, so could "except each"
> ("excluding each of these") and "except group" ("excluding this group").
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FUTYE36MLFAWU72KTHDEQY5JFDA2PQ4G/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/K3I7552LB6O5XS3KWLC2C2U5XME43HQ5/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 654 except* formatting
We wonder if people have a view on which of the following is clearer/better: 1. except *E as e: // except *(E1, E2) as e: 2. except* E as e: // except* (E1, E2) as e: (The difference is in the whitespace around the *). At the moment * is a separate token so both are allowed, but we could change that (e.g., make except* a token), and in any case we need to settle on a convention that we use in documentation, etc. It is also not too late to opt for a completely different syntax if a better one is suggested. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4B256YKUPW5P2M44GG5H6FBL3PSV6ODP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 663: Improving and Standardizing Enum str(), repr(), and format() behaviors
> On 11 Sep 2021, at 06:57, Ethan Furman wrote: > > If the previous output needs to be maintained, for example to ensure > compatibily between different Python versions, software projects will need to > create their own enum base class with the appropriate methods overridden. Perhaps you could provide that backwards-compatible base class, either in the library as a deprecated class or at least in the PEP for people to copy if they need it? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YFKBT3FTNRZHTBWT6CUV62CUKSHVTTYK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
On Thu, Aug 19, 2021 at 11:28 AM Duncan Grisby > Thanks. Victor Stinner has now commented on the defect I raised to say > that it is actually a bug in the traceback module, in that it should not be > using Py_DECREF at all inside a signal handler. The fact that it leads to > use of the old trashcan macros is not the cause of the crash that I saw. > > I therefore don't think it's necessary for me to check with the updated > frameobject.c. Do you agree? I can try to reproduce the situation, but I > can't just quickly test it because the infrastructure where the crash was > seen is now being used for other things. > Well, in order to verify that we fixed the problem you reported we need to be able to test the fix and see that the problem is gone. Right now we have two theories of what the cause could be, because there are two known bugs in that area. But you may have stumbled upon an unrelated third bug. The best scenario is that we close a bpo issue because we can reproduce the bug and see that it has been fixed. When that's not possible, there will come a point where we will close it and say "the code has changed since you observed the problem and we fixed things that could have caused it. Please open a new issue if you still see this bug". ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/U67YN4TGYMB2ROSFRX34QAGMYYGYDLBN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
Thanks, I've updated the PR. https://github.com/python/cpython/pull/27693 I added migration instructions to the news file (which Pablo can copy into what's new once it's committed). Also added a Py_DEPRECATED(3.11) typedef which is used in the macro. It remains the decide how to backport the compiler warning to older versions, and to which versions. On Tue, Aug 17, 2021 at 5:18 PM Victor Stinner wrote: > If the macro is deprecated, please provide an detailed explanation how > to port existing C code to the new C API, in What's New In Python 3.X > (version where the macro is deprecated, Python 3.11 if I understood > correctly). ... ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PJQMYGD567MPZEBPPVD3RX6ATDCSPJCA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
On Tue, Aug 17, 2021 at 12:05 PM Duncan Grisby wrote: > I don't know if it is pertinent to this at all, but I raised > https://bugs.python.org/issue9 in which the faulthandler module can > lead to a segfault inside Py_TRASHCAN_SAFE_BEGIN. Would that be avoided if > frameobject.c was changed to use Py_TRASHCAN_BEGIN / END? > We just changed frameobject.c to use the new macros. Can you check? https://github.com/python/cpython/pull/27683 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4HBXLRBGNANOUMWTABWU7MPJZL5ML6V6/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Problems with dict subclassing performance
This is the source of this whole misunderstanding: On Sun, Aug 15, 2021 at 9:34 PM Marco Sulla wrote: > > I asked for help, because I didn't know __nothing__ about what Monica > replied to me in the SO answer. And you say to me my post seems spam? > > [snipped] > And you continue to mark my post as > spammy, when I simply tried to get help! This is unacceptable! > Marco thought this was a help-with-python list, and in this light his reaction sort of makes sense. The reality, Marco, is that this list is for discussions about python development, not for seeking help with understanding python. It is not appropriate to "ask for help" on this list, and you are not entitled to a reply within X working days, or ever. People will engage with what you write if they want to. There are plenty of help-with-python forums. Good luck. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/56QZTQH557B7BNFBIDJC7XKMUVQL3DC5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Towards removing asynchat, asyncore and smtpd from the stdlib
The asynchat [1] and asyncore [2] libraries have been deprecated since 3.6 in favor of asyncio [3]. In addition, smtpd [4], which depends on asynchat and asyncore, is deprecated in favor of aiosmtpd [5]. The documentation for each of these libraries mentions that it is deprecated and refers to the suggested replacement. However, they do not yet emit DeprecationWarnings at import, so people may be using them without being aware that they are no longer maintained (there are a few dozen open issues on bpo, some of them reporting bugs). The next step is to add deprecation warnings, so that we can eventually delete them. There is also the issue that some of the stdlib tests are still using these libraries, but this does not need to block removing them from the stdlib because we can move them to test.support until those tests are rewritten. Note that nobody needs to stop using these libraries if they don't want to migrate to the suggested alternatives - the modules are pure python and anyone can include a copy in their own codebase and continue using it in a legacy application (or even continue maintaining it if they want). Barry and I are working on a patch to add deprecation warnings in 3.10 when one of these are imported [6]. Let us know if you have any comments on this plan. [1] https://docs.python.org/3/library/asynchat.html [2] https://docs.python.org/3/library/asyncore.html [3] https://docs.python.org/3/library/asyncio.html [4] https://docs.python.org/3/library/smtpd.html [5] https://aiosmtpd.readthedocs.io/en/latest/ [6] https://github.com/python/cpython/pull/26882 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EN7X7OM63PE2FKVYGVGQE44YKBVDICSU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Roundup to GitHub Issues migration
On Tue, Jun 22, 2021 at 11:22 PM Terry Reedy wrote: > > With 2.x gone and a backport flow, essentially all issues are for main, > so leave version tags off the issue (and eliminate the need to update > them!). For enhancement requests I agree we probably don't need to keep updating the version (though it might be useful to know which version the user was looking at when coming up with the suggestion). But I think for a bug report it's useful to know the most recent version on which the bug was reproduced. A large part of triage work is to check whether a bug report is still relevant. If you see "version 3.5" then it makes sense to check on a current version and either update the issue or close it. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2UWBN6METJ3WRJJF2ZCQDCHTGOPNQC7V/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: name for new Enum decorator
Andrei is suggesting to look at each enum value as the set of "bits set to 1" in this value, and then apply a set-thoery term to the problem. A set of "cardinality 1" in this context is an enum value with only one "1", what you called canonical. The condition you defined is that any bit that is set anywhere, is also in a canonical value. Anyway, I don't know whether this kind of terminology is widely accessible. Maybe you could use something like @composite. As in, every non-canonical value in the enum is the composition of canonical values. On Mon, Jun 7, 2021 at 6:36 PM Ethan Furman wrote: > On 6/6/21 9:14 AM, Irit Katriel via Python-Dev wrote: > > On 6 Jun 2021, at 16:58, Andrei Kulakov wrote: > > >> In Math / CompSci there is a definition that almost exactly matches > this: Exact Cover - > >> https://en.wikipedia.org/wiki/Exact_cover > >> > >> The difference is that, IIRC, solving the problem is finding and > removing all subsets that are unneeded to create an > >> exact cover, so it's kind of arriving at it from a different > direction, but 'exact cover' definition itself is a good > >> match. > > > > I’m not sure it’s a quite the same - it doesn’t require that the sets > in the cover have cardinality 1, which I think > > Ethan does. > > Well, I'm not sure what "cardinality 1" means, so I don't know if I do or > not. :) > > -- > ~Ethan~ > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/BMCG6MA3HKYAQC4JARJFXZDOZGCSIPMM/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2OKVXULZMCEMWUNAMT4ENQVQP2JBSLDV/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: name for new Enum decorator
> On 6 Jun 2021, at 16:58, Andrei Kulakov wrote: > > > In Math / CompSci there is a definition that almost exactly matches this: > Exact Cover - https://en.wikipedia.org/wiki/Exact_cover > > The difference is that, IIRC, solving the problem is finding and removing all > subsets that are unneeded to create an exact cover, so it's kind of arriving > at it from a different direction, but 'exact cover' definition itself is a > good match. I’m not sure it’s a quite the same - it doesn’t require that the sets in the cover have cardinality 1, which I think Ethan does. And the ‘exact’ requirement (that each bit is covered once) surely doesn’t apply. It’s more like “full cover by singletons”. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AKFYNL5DFLCKTLIHCDSLNPAP3GQDKVEY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Proposal: declare "unstable APIs"
Maybe ‘version-dependent’ api? It indicates why the api is unstable (as opposed to something like test.support where the docs say ‘this is for us, we’re not bothered about keeping it stable’). In some contexts ‘unstable’ means buggy/unreliable. > On 4 Jun 2021, at 00:20, Glenn Linderman wrote: > > > On 6/3/2021 3:34 PM, Guido van Rossum wrote: >> On Thu, Jun 3, 2021 at 3:17 PM Tim Delaney >> wrote: >>> Perhaps "living API" analogous to "living document". Much more positive >>> connotations ... >> >> Perhaps, but that's pretty much coining a new term, which we would then have >> to explain. And since the opposite would be "dead API", what is a dead API >> exactly? And doesn't it follow that all APIs are living? If we stick with >> unstable, we're adopting a term that's in use by at least one other language >> community (Rust), and the slightly negative connotation is welcome -- people >> should think twice before using unstable APIs. > > I read somewhere that the term "stable" means "dead" in some contexts... was > it maybe a medical context? So "living" would be "unstable" too, as Tim > suggested. > > And since people know what a living document is, a living API wouldn't be > much of a stretch. > > On the other hand, "unstable" carries a bit more connotation of "needs > caution". > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/CS23XQX3356Q3S6WI66PISIT3NXEGMGM/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NBODHC6YELGWHLBRHJCJPINPZT6OZD4F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The repr of a sentinel
If we drop the requirement for pickle round-tripping then I would add a requirement that sentinel is unpicklable, to prevent accidents. Irit On Fri, May 14, 2021 at 8:38 PM Tal Einat wrote: > > I'll try to organize my thoughts a bit here. This is a bit long, > welcome to skip to the final sentence for the "tl;dr". > > Features one may want for a sentinel: > 1. Unique from other objects > 2. Globally unique, i.e. unique from other such sentinels (no consensus > here) > 3. Clear repr (the original subject of this thread) - significant > since these often appear in function signatures > 4. Survives pickling round-trip (I brought this up since I was bitten > by this once, but others have mentioned that this is usually > irrelevant) > 5. Can be given a clear type signature (this was brought up on > twitter[1]) - significant since without this nobody can add full type > signatures even if they want to > > The common `SENTINEL = object()` idiom fails #3, #4 and #5. This is > what I've been using for years, and I now think that it isn't good > enough. This not having a nice repr is what started this thread. > > I'd also personally prefer something simple, ideally without a new > class or module. > > There are several simple idioms using existing features that seem like > good options, so let's review those: > > 1. Ellipsis, a.k.a. `...`. This has all of the features outlined above > except #2. My main issue with using Ellipsis for this is that it could > be surprising and confusing for devs first encountering such use, and > could be relatively awkward to figure out. > > 2. An instance of a one-off class. dataclasses.MISSING is an example > of this. It is defined thus: > > class _MISSING: > pass > MISSING = _MISSING() > > Besides failing #4 (surviving pickle round-trips), its repr isn't > great: . That is easily > overcome by implementing __repr__. > > 3. A one-off class: > > class MISSING: pass > > This has all of the above features except #5: having a clear type > signature (since its type is type). Using a class as a value this way > could be surprising, though. It's repr also isn't great: 'dataclasses._MISSING'>. > > 4. A value of an single-valued enum, for example (from [1]): > > class _UNSET(enum.Enum): > token = enum.auto() > > This has all of the features above and is simple, just requiring a > comment to explain what it is. It's repr is a bit awkward though: > > >>> repr(_UNSET.token) > '<_UNSET.token: 1>' > > > All of these are in use by some developers, though not necessarily in > the stdlib. None is perfect, though all are probably good enough. > Since pickling is likely not relevant in most cases, I'm currently in > favor of #2 making sure to implement a nice __repr__. > > - Tal > > [1] https://twitter.com/nolar/status/1392962447166877697 > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/BYLTDD72RZUUKVRJL6TTSWF35JD3FL47/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7I7S22KEOZY2MDDIYCUPQWG6WXLFZZR5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: The repr of a sentinel
On Thu, May 13, 2021 at 10:28 AM Antoine Pitrou wrote: > > I agree that is a reasonable spelling. > > I initially suggested , but now I'm not sure because it doesn't indicate what happens when you don't provide it (as in, what is the default value). So now I'm with or . The arg is only there for backwards compatibility now. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/C5W6NT6KYF2M2QW2WZZ6V33KTMIVICDY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] The repr of a sentinel
Following a recent change, we now have in traceback.py: _sentinel = object() def print_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None, file=None, chain=True): So now: >>> import traceback >>> help(traceback.print_exception) Help on function print_exception in module traceback: print_exception(exc, /, value=, tb=, limit=None, file=None, chain=True) Is there a convention on how such default sentinel values should appear in docs? https://bugs.python.org/issue43024 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZLVPD2OISI7M4POMTR2FCQTE6TPMPTO3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
Another alternative is instead of File blah.py line 3: return x/0 ^^^ to have File blah.py line 3 cols 12-14: x/0 On Mon, May 10, 2021 at 11:12 AM Steven D'Aprano wrote: > On Mon, May 10, 2021 at 05:34:12AM -0400, Terry Reedy wrote: > > On 5/10/2021 3:28 AM, M.-A. Lemburg wrote: > > > > >I'm mostly thinking of tracebacks which go >10 levels deep, which is > > >rather common in larger applications. For those tracebacks, the top > > >entries are mostly noise you never look at when debugging. The proposal > > >now adds another 10 extra lines to jump over :-) > > > > If the slice were instead marked with color tagging, as I hope will be > > possible in IDLE and other IDEs, then no extra lines well be needed > > That's great for people using IDLE, but for those using the vanilla > Python interpreter, M-A.L makes a good point about increasing the > vertical size of the traceback which will almost always be ignored. > > Its especially the case for beginners. Its hard enough to get newbies to > read *any* of the traceback. Anything which increases the visual noise > of that is going to make it harder. > > > -- > Steve > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/3ADVDPF4Z5DMXKG2CMJ3JTIN2SC76AUC/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YWNDAKU7RYBFWCRMXK3UZEYFE7NLUSNY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks
On Fri, May 7, 2021 at 10:52 PM Pablo Galindo Salgado wrote: > > The cost of this is having the start column number and end column number > information for every bytecode instruction > Is it really every instruction? Or only those that can raise exceptions? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UL5XAR2MGYJPIKB67R56OJXUVKP2KM3H/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Nathaniel, Philosophy of Science 101: nobody knows what TheBestSolution is and nobody ever will. That said, when we decided to submit PEP 654, that was a point in time when we came to believe that we found the right solution. We made significant changes later following feedback we agreed with, but in the case of "flat EG" I see several disadvantages and don't quite understand what the win is. I appreciate the invitation, but I don't want to work on that idea (nor am I the right person to push it forward). However, I think it's fantastic if you and others do pursue alternative options (independently of my biases). You will either find something better or validate PEP 654, and in either case the world wins. Feel free to adapt our implementation if that helps. You have a year before we have our next chance to commit to PEP 654. Irit On Thu, May 6, 2021 at 10:18 AM Nathaniel Smith wrote: > On Thu, Apr 29, 2021 at 9:14 AM Yury Selivanov > wrote: > > Nathaniel, at this point it's clear that this thread somehow does not > > help us understand what you want. Could you please just write your own > > PEP clearly outlining your proposal, its upsides and downsides? > > Without a PEP from you this thread is just a distraction. > > If that's the best way to move forward, then ok. My main thing is just > that I don't want to make this some antagonistic me-vs-you thing. > After all, we all want the best design to be chosen, and none of us > know what that is yet, so there's no need for conflict :-). > Irit, Yury, would you be interested in co-authoring a PEP for the > "flat EG" approach? Basically trying to set down the best possible > version of each approach, so that we can put them next to each other? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BOSMGZQHX3OHXEVRMEJXY6FONIJT7MNP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python release timeline plot
This is just what I needed, thanks!
I’m in the process of preparing a presentation for my team at Bank of America
about how the dev lifecycle of our system (Quartz risk platform) interplays
with the dev lifecycle of Python (and how we can do better). I will use this.
> On 2 May 2021, at 08:33, Michał Górny wrote:
>
> Hi,
>
> I've been working on visualizing some data on Python in Gentoo, and made
> a timeline of Python releases as a side result. I've figured some of
> you might be interested, so I've decided to share it. The result:
>
> https://mgorny.pl/python-timeline.html
>
> Script: https://github.com/mgorny/gpyutils/blob/master/timeline.py
> Data: https://github.com/mgorny/gpyutils/blob/master/timeline.toml
>
> I've (roughly) covered the versions between 2.5 and 3.10 (2.5 is
> the oldest I've seen during my time in Gentoo, also the timeline's
> gotten narrow already).
>
> dev covers the period between start of development and first alpha.
> alpha goes from a1 to b1, beta from b1 to rc1, rc from rc1 to final.
> stable goes to the last bugfix release, and security from that until
> the last security release.
>
> A few notes:
>
> 1. The data is cut off to the right, and for technical reasons this
> means tooltips will display early end dates.
>
> 2. The right-most data is based on approximate dates where schedules
> aren't precise, esp. 3.10 assumes 'approximately 18 months' for stable
> status.
>
> 3. Start of development is only mentioned in the newest schedules. For
> older versions, I've took the data of the commit changing PYTHON_VERSION
> in configure.{ac,in} (which seems to match start of development in newer
> schedules).
>
> 4. Where data on last bugfix release wasn't available in schedules, I've
> looked through release notes.
>
> 5. I've deliberately skipped dev period for 2.5 not to overstretch it to
> the left.
>
> If you see any mistakes, please let me know.
>
> --
> Best regards,
> Michał Górny
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CLSHKQAFYXYPMRX7J6RWUS2XQSRMIBUN/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/UJFL2SIBSOI6HPY4Y32WG5P7TI4FB6RT/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?
Re https://bugs.python.org/issue40608. I think it will be an act of kindness to deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10 and tell people to use Py_TRASHCAN_BEGIN/END instead. TL;DR: There was a change in 3.8 that introduced the latter while leaving the former for backwards compatibility, but also inadvertently breaking them. This is not an easy bug to deal with in the wild, we found it because we have a unit test in our codebase referencing https:// bugs.python.org/issue16602. A deprecation note pointing to the new macros would have made it easier. Is there any reason not to deprecate the old macros? Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LWP6MOLP5UW2TH3MROZQK4N64SX35N2B/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
On Fri, Apr 23, 2021 at 9:22 AM Nathaniel Smith wrote: > On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum wrote: > > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith wrote: > >> Sure. This was in my list of reasons why the backwards compatibility > >> tradeoffs are forcing us into awkward compromises. I only elaborated > >> on it b/c in your last email you said you didn't understand why this > >> was a problem :-). And except* is definitely useful. But I think there > >> are options for 'except' that haven't been considered fully. > > > > Do you have any suggestions, or are you just telling us to think harder? > Because we've already thought as hard as we could and within all the > constraints (backwards compatibility and otherwise) we just couldn't think > of a better one. > > The main possibility that I don't think we've examined fully is to > make 'except' blocks fire multiple times when there are multiple > exceptions. We ruled it out early b/c it's incompatible with nested > EGs, but if flat EGs are better anyway, then the balance shifts around > and it might land somewhere different. it's a tricky discussion > though, b/c both the current proposal and the alternative have very > complex implications and downsides. So we probably shouldn't get too > distracted by that until after the flat vs nested discussion has > settled down more. > > I'm not trying to filibuster here -- I really want some form of EGs to > land. I'm very glad to hear that. It's been hard to know where you stand, because you didn't even decline our invitation in October to work together on this, and several later invitations to look at the implementation and try it with Trio -- you just didn't reply. The only responses I saw were public, on this list, taking us right back to the drawing board (for example - the suggestion you mention in the previous paragraph as not-sufficiently-explored, actually appears in the rejected ideas section of the PEP, and the reason for rejection is not that it's incompatible with nesting). So thank you for clarifying that you are, ultimately, supportive of our efforts. We do realize that we didn't adequately credit you for the contributions of your 2018 work to this PEP, and have now added an acknowledgements section for that purpose. Apologies for that. I'm confused about the flattening suggestion - above you talk about "flat EG", but below about tracebacks. It's not clear to me whether you want EG to be flat (ie no nesting of EGs) or just the traceback to be flat (but you can still have a nested EG). I also don't know what problem you are trying to solve with this. Fortunately we're not at the whiteboard stage anymore, we have a fully working implementation and you can use it to demonstrate any problems you see with what we did. Irit > I think python has the potential to be the most elegant and > accessible language around for writing concurrent programs, and EGs > are a key part of that. I don't want to fight about anything; I just > want to work together to make sure we have a full picture of our > options, so we can be confident we're making the best choice. > > > The real cost here is that we would need a new "TracebackGroup" concept, > since the internal data structures and APIs keep the traceback chain and > the exception object separated until the exception is caught. In our early > design stages we actually explored this and the complexity of the data > structures was painful. We eventually realized that we didn't need this > concept at all, and the result is much clearer, despite what you seem to > think. > > I'm not talking about TracebackGroups (at least, I think I'm not?). I > think it can be done with exactly our current data structures, nothing > new. > > - When an EG is raised, build the traceback for just that EG while > it's unwinding. This means if any C code peeks at exc_info while it's > in flight, it'll only see the current branch of the traceback tree, > but that seems fine. > - When the exception is caught and we go to write back the traceback > to its __traceback__ attribute, instead "peek through" the EG and > append the built-up traceback entries onto each of the constituent > exceptions. > > You could get cleverer for efficiency, but that basic concept seems > pretty simple and viable to me. What am I missing? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DO3ESI4RRXHSY7C3YC5VAEPN3XNN5XIY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
On Wed, Apr 21, 2021 at 11:22 PM Nathaniel Smith wrote: > > > Saying that you have to make a new API every time you start using > concurrency inside a function is extremely restrictive. You don't need a new API when you start using concurrency inside a function. You need a new API when you start raising exception groups from a function because that changes the function's external behaviour. -1 from me on the denormalized traceback idea. I've said what I had to say about it in my previous email. Cheers Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HVPVPRSTJMI7KOFXQLJJCRUPC2R4IE42/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
I don’t see what this simplifies. We still need to implement split, and to worry about wrapping or not wrapping BaseExceptions and we still need to define exception handling semantics (except/except*). > On 20 Apr 2021, at 22:12, srku...@mail.de wrote: > > So, forgive me my relatively simple mental model about ExceptionGroup. I > still try to create one for daily use. > > As noted in the discussion, an EG provides a way to collect exceptions from > different sources and raise them as a bundle. They have no apparent relation > up until this point in time (for whatever reason they have been separate and > for whatever reason they are bundled now). The result would be a tree graph > in any case. > > A usual datastructure for a tree is to store all child nodes at the parent > node. > > That was the idea behind the content of BaseException.__group__: it’s the > list of child exceptions bundled at a specific point in time and raise as > such a bundle. So all exceptions could become EGs with the additional > semantics you‘ve described in the PEP. > > Illustrative Example: bundle_exc.__group__ > [IOError(123), RuntimerError(‘issue somewhere’)] > > I was wondering what of the PEP could be removed to make it simpler and more > acceptable/less confusing (also looking at reactions from Twitter etc.) and I > found these additional classes to be a part of it. Additionally, I fail to > see how to access these bundled exceptions in an easy manner like __cause__ > and __context__. (As the PEP also referring to them). So, I removed the > classes and added a regular attribute. > > The reason I brought this up what the section “rejected ideas” didn’t showed > anything in this direction (or I managed to missed that). > > Sven > >> >> On 20. Apr 2021, at 22:05, Irit Katriel wrote: >> >> Hi Sven, >> >> I don’t follow. What would the value of __group__ be and how would it work? >> >> Irit >> On 20 Apr 2021, at 20:44, srku...@mail.de wrote: >>> >>> Hi Irit, >>> reading this subthread specifically, I just got a wild idea and I couldn‘t >>> find any related information in the PEP: >>> Why not extending BaseException by __group__ among __cause__ and >>> __context__? >>> Would this reduce some of the added complexity and thus increase broader >>> acceptance? >>> Cheers, >>> Sven > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OJJXMPBD4ZUZOUZ2ULWT4Q6ULI75HLDE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Sven, I don’t follow. What would the value of __group__ be and how would it work? Irit > On 20 Apr 2021, at 20:44, srku...@mail.de wrote: > > > Hi Irit, > > reading this subthread specifically, I just got a wild idea and I couldn‘t > find any related information in the PEP: > > Why not extending BaseException by __group__ among __cause__ and __context__? > > Would this reduce some of the added complexity and thus increase broader > acceptance? > > Cheers, > Sven ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NYAXQQ7EVXUNX7MTQJDLAXVKGRHAUSEF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
On Tue, Apr 20, 2021 at 2:48 AM Nathaniel Smith wrote:
>
> The problem is that most of the time, even if you're using concurrency
> internally so multiple things *could* go wrong at once, only one thing
> actually *does* go wrong. So it's unfortunate if some existing code is
> prepared for a specific exception that it knows can be raised, that
> exact exception is raised... and the existing code fails to catch it
> because now it's wrapped in an EG.
>
Yes, this was discussed at length on this list. Raising an exception group
is an API-breaking change. If a function starts raising exception groups
its callers need to be prepared for that. Realistically we think exception
groups will be raised by new APIs. We tried and were unable to define
exception group semantics for except that would be reasonable and backwards
compatible. That's why we added except*.
> > It is easy enough to write a denormalize() function in traceback.py that
> constructs this from the current EG structure, if you need it (use the
> leaf_generator from the PEP). I'm not sure I see why we should trouble the
> interpreter with this.
>
> In the current design, once an exception is wrapped in an EG, then it
> can never be unwrapped, because its traceback information is spread
> across the individual exception + the EG tree around it. This is
> confusing to users ("how do I check errno?"), and makes the design
> more complicated (the need for topology-preserving .split(), the
> inability to define a sensible EG.__iter__, ...). The advantage of
> making the denormalized form the native form is that now the leaf
> exceptions would be self-contained objects like they are now, so you
> don't need EG nesting at all, and users can write intuitive code like:
>
> except OSError as *excs:
> remainder = [exc for exc in excs if exc.errno != ...]
> if remainder:
> raise ExceptionGroup(remainder)
>
We have this precise example in the PEP:
match, rest = excs.split(lambda e: e.errno != ...)
You use split() instead of iteration for that. split() preserves all
__context__, __cause__ and __traceback__ information, on all leaf and
non-leaf exceptions.
> For display purposes, it is probably nicer to look at a normalized
> traceback where common parts are not repeated.
>
> Yeah, I agree; display code would want to re-normalize before
> printing. But now it's only the display code that needs to care about
> figuring out shared parts of the traceback, rather than something that
has to be maintained as an invariant everywhere.
>
If you *do* want iteration, we show in the PEP how to write a
leaf_generator() that gives you the leaf exceptions with their tracebacks
(as a list of chunks). It is easy to copy the chunks into a single flat
traceback. We didn't propose to add it to traceback.py yet because the use
case is unclear but if people need it we're talking about 10-15 lines in
traceback.py.
So for your suggestion:
Pros:
1. Those who want a denormalized traceback (if they exist) won't need to
call traceback.denormalize().
Cons:
1. A significant change in the interpreter that will make it less efficient
(both time and space).
2. Display code will need to normalize the traceback, which is much more
complicated than denormalizing because you need to discover the shared
parts.
Am I missing something?
>
> > It sounds like you want some way to enrich exceptions. This should be
> optional (we wouldn't want EG to require additional metadata for
> exceptions) so yeah, I agree it should sit on the leaf exception and not on
> the group. In that sense it's orthogonal to this PEP.
>
> Well, the extra metadata would specifically be at "join" points in the
> traceback, which are a thing that this PEP is creating :-). And it's
> useful for every user of EGs, since by definition, an EG is
> multiplexing exceptions from multiple sources, so it's nice to tell
> the user which sources those are.
That said, you're right, if we want to handle this by defining a new
> kind of traceback entry that code like Trio/asyncio/hypothesis can
> manually attach to exceptions, then that could be written as a
> separate-but-complementary PEP. In my original design, instead of
> defining a new kind of traceback entry, I was storing this on the EG
> itself, so that's why I was thinking about it needing to be part of
> this PEP.
>
You can also create an ExceptionGroup subclass with whatever extra data you
want to include.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PTHRBLGCBGZTGIMCTIB5HDQHPXFUCNLQ/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Thomas and the rest of the SC, Thank you for updating us and for your kind words about the PEP. We agree that it is safer to include this in 3.11 and not rush it into 3.10. As you say, having time to develop the integration with asyncio before finalizing the API will give us more confidence in the design. We would like to commit the code into the 3.11 development branch soon, as you suggest. We will be looking forward to the (provisional?) acceptance of the PEP by the SC so that we can do this. Best regards Irit, Yury & Guido On Thu, Apr 15, 2021 at 6:48 PM Thomas Wouters wrote: > > Irit, Yury and Guido, > > Thanks for submitting PEP 654 (Exception Groups and except *). The > Steering Council isn’t done debating this PEP, but we wanted to give an > update so everyone knows what to expect. > > We are mostly in favour of the PEP, but we’re worried about the timing -- > not just because there is still some discussion going on, but also the time > available for us to discuss the proposal, and to validate the new API once > it’s accepted. We are rapidly approaching feature freeze for Python 3.10, > and while we’re not particularly worried about the implementation of this > PEP, we would like to see actual usage of the new API with enough time to > tweak it before the official release. That is to say, we would like to see > the "analysis of how exception groups will likely be used in asyncio > programs" turn into "here is how exception groups are used in asyncio > programs" before the final release, like in the asyncio module and the > standard library (for example, with TaskGroups). There’s also a > presentation scheduled in the PyCon US Language Summit, which may lead to > more discussion of the PEP and perhaps evolution of its ideas. > > What would you and others feel about postponing the PEP until Python 3.11, > getting it in as soon as the branch opens, so that potential users of the > API -- asyncio or otherwise -- can provide us with practical experience to > validate some of the assumptions made in the proposal, and to discover any > warts or pitfalls we missed or misjudged? I don’t mean that as a slight on > the PEP, which is thoughtful, thorough and excellently written, but as a > practical observation of the complexity involved. Most of the PEP is more > of a building block for other APIs than something many users will be > directly exposed to, and we want to make sure the end result isn’t affected > by unanticipated effects of these design decisions. (This would potentially > also give third-party users more time to evaluate the API, although we know > adoption of alpha releases isn’t big enough to make this a very strong > argument for waiting.) > > Like I said, we’re still discussing the PEP, so this hasn’t been decided. > If you or others feel strongly that the benefit of postponing is too small > (or the cost too high), we’ll take that into consideration. The SC has been > somewhat busy of late, and we don’t want to let our schedule be the > deciding factor here, but this may come down to there just not being enough > time before the feature freeze to make a thoughtful decision. > > For the SC, > Thomas. > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WSXKY6FVJ7EFQ533OJE67ZIISAAX2QJE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
On Mon, Apr 5, 2021 at 2:59 PM Chris Jerdonek
wrote:
> This point reminded me again of this issue in the tracker ("Problems with
> recursive automatic exception chaining" from 2013):
> https://bugs.python.org/issue18861
> I'm not sure if it's exactly the same, but you can see that a couple of
> the later comments there talk about "exception trees" and other types of
> annotations.
>
> If that issue were addressed after ExceptionGroups were introduced, does
> that mean there would then be two types of exception-related trees layered
> over each other (e.g. groups of trees, trees of groups, etc)? It makes me
> wonder if there's a more general tree structure that could accommodate both
> use cases...
>
> --Chris
>
Interesting, I commented on that issue - I think we may be able to solve it
without adding more trees.
That said, we will have groups-of-trees/trees-of-groups. Already today, an
exception plus its chained __cause__s and __context__s is the root of a
binary tree of exceptions. The nodes of this tree represent the times that
the exceptions were caught.
An exception group is a tree where the nodes represent the times when
exceptions were grouped together and raised.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BZWELKDUAKCOXSH5KQRFGQJRQWJ2OHKW/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Nathaniel, Thank you for your feedback. See a few comment below. On Mon, Apr 5, 2021 at 11:01 AM Nathaniel Smith wrote: > OK, better late than never... here's a much-delayed review of the PEP. > Thank you Irit and Guido for carrying this forward while I've been AWOL! > It's fantastic to see my old design sketches turned into something like, > actually real. > > == Overall feelings == > > Honestly, I have somewhat mixed feelings ExceptionGroups. I don't see any > way around adding ExceptionGroups in some form, because it's just a fact of > life that in a concurrent program, multiple things can go wrong at once, > and we want Python to be usable for writing concurrent programs. Right now > the state of the art is "exceptions in background threads/tasks get dropped > on the floor", and almost anything is better than that. The current PEP is > definitely better than that. But at the same time, there are a lot of > compromises needed to retrofit this onto Python's existing system, and the > current proposal feels like a bunch of awkward hacks with hacks on top. > That's largely my fault for not thinking of something better, and maybe > there is nothing better. But I still wish we could come up with something > more elegant, and I do see why this proposal has made people uncomfortable. > People were uncomfortable about three main points: 1. In the first draft "except Exception" would not catch exception groups, this was changed in the second draft. 2. In the first draft exception groups were not subclassable, this also changed in the second draft. 3. Do we really need except*? Why is ExceptionGroup not enough? I think we replied, but this question may still be on some people's minds and if so we should continue discussing it. Was there anything else? > For example: > > - I'm uncomfortable with how in some contexts we treat EG's as > placeholders for the contained exceptions, and other places we treat them > like a single first-class exceptions. (Witness all the feedback about "why > not just catch the ExceptionGroup and handle it by hand?", and imagine > writing the docs explaining all the situations where that is or isn't a > good idea and the pitfalls involved...) If we could somehow pick one and > stick to it then I think that would make it easier for users to grasp. (My > gut feeling is that making them pure containers is better, which to me is > why it makes sense for them to be @final and why I keep hoping we can > figure out some better way for plain 'except' and EGs to interact.) > I'm not sure what you mean by "a placeholder". An exception group is an exception, and except* is a new syntax that helps you manipulate exception groups. I don't think the confusion you mention was due to the fact that except can catch EGs, but rather due to the fact that the simplest examples don't show you why except is not good enough, and why we need except*. Suppose we did as you suggest and made exception group a container which is not exception. Now suppose that an exception is raised in an except* block. What is the context of this exception? Do we change the requirement that an exception's context is always an exception? How do you raise an exception group if it's not an Exception? Do we go back to allowing random objects being raised? > > - If a function wants to start using concurrency internally, then now > *all* its exceptions have to get wrapped in EGs and callers have to change > *all* their exception handling code to use except* or similar. You would > think this was an internal implementation detail that the caller shouldn't > have to care about, but instead it forces a major change on the function's > public API. And this is because regular 'except' can't do anything useful > with EGs. > I don't understand this point. If you are using concurrency internally and don't want to raise EGs externally, then surely you will catch EGs, select one of the exceptions to raise and throw away all the others like in the old days when there weren't EGs (which is presumably what your caller is expecting). In other words, if you make drastic changes in a function's implementation, it is your responsibility to ensure that the old API is preserved. > > - We have a special-case hack to keep 'except Exception' working, but it > has tricky edge cases (Exceptions can still sneak past if they're paired up > with a BaseException), and it really is specific to 'except Exception'; it > doesn't work for any other 'except SomeError' code. This smells funny. > I agree with this point, my personal preference would be to have ExceptionGroup(BaseExceptionGroup) as in the first draft, with a runaway exception group being something you should fix. The current choice is a compromise for backwards compatibility, not so much with the language but with the practice of using "except Exception" to "catch almost everything, log it and move on". I don't think we can go against that at this point. > > Anyway, that's
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Greg, If all you want is to catch an exception group and process it, then except* does look like overkill. It gets more interesting if you want to handle only some of the exceptions and reraise the rest (without adding the current frame to the traceback), or when the exception handler raises/reraises exceptions (with 'raise e' or bare 'raise'). We can look to Trio to see what exception handling would be like if we add ExceptionGroup without except*. Trio's exception group is called MultiError, and this part of the tutorial explains how they can be handled: https://trio.readthedocs.io/en/stable/reference-core.html#trio.MultiError So asyncio (or some new module in the stdlib) would probably end up exposing something along the lines of MultiError.catch, a context manager where you register the exception handler as a callback and the unhandled exceptions are automatically reraised. You can see the trio code for that here: https://github.com/python-trio/trio/blob/master/trio/_core/_multierror.py#L129 The trick it does to reset __context__ in the finally block won't work for the __traceback__ (I tried). So it always adds the current frame (if it doesn't return before line 145). Also, it doesn't seem to support raising exceptions from the handler: https://github.com/python-trio/trio/blob/6754c74eacfad9cc5c92d5c24727a2f3b620624e/trio/_core/_multierror.py#L79 So yes, we can catch exception groups and handle them like normal exceptions, but I don't see a clean way to catch parts of them selectively or to raise exceptions from exception handling code. Irit On Sat, Mar 27, 2021 at 10:57 PM Greg Ewing wrote: > While we're talking about compelling use cases, does anyone have an > actual, concrete use case for the proposed "except *" feature that's > strong enough to justify new syntax? > > I'm fine with having ExceptionGroup as a built-in type. I'm not fine > with adding new syntax that will apparently be used only in rare > circumstances. > > Can code that's aware of the possibility of getting an ExceptionGroup > not simply catch it as a normal exception and then pick it apart? Do > we really need a whole new piece of machinery for this? > > -- > Greg > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/EIW6JGXRWKEEIY72CGZ7BMKU3LHJIZQK/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WYY6RTRNIP5ZV6OJVNZB3QZL3LWNQKT5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Paul,
On Sat, Mar 27, 2021 at 6:00 PM Paul Sokolovsky wrote:
>
> It definitely feels like a lot of effort went into devising and
> polishing ExceptionGroup's and except*, thanks. But I'm not sure if you
> gentlemen come up with the "ultimate" way to deal with multiple errors,
>
I've been mistaken for a man before, but no-one has ever confused me for
gentle. I'll take that as a compliment.
> which deserves being codified on the language level (vs be added as
> pluggable means indeed).
Pluggable is not without its problems. I'm all in favor of you developing
this idea and proposing an alternative.
As I said before, you just need to answer two questions:
1. show a limitation of our approach (the contrived flat-set is not one -
see below)
2. describe how a pluggable approach works for that case
> > > > def chained(e, excs):
> > > > while e:
> > > > if isinstance(e, excs):
> > > > return e
> > > > e = e.__cause__ # Simplified, should consider __context__
> > > > too
> > > >
> > > > try:
> > > > tempfile.TemporaryDirectory(...)
> > > > except *chained(OSError) as e:
> > > > print(e)
> >
> > What if the user code raised an OSError too, so now that exception
> > group has more than one?
>
> I don't see how that's different from PEP654's ExceptionGroup's case.
> It's just in ExceptionGroup, there would be 2 (unrelated?) OSError's,
> while using normal chaining rules, there's at least context info which
> is preserved.
>
It's different because your "match" function returns a single exception
(the first one that is of OSError type). Any further OSErrors will be
reraised. The PEP's except* knows how to match multiple exceptions of the
relevant type.
>
> > I am interested to know (1) what limitations you see in the
> > ExceptionGroup type we are proposing (in what sense is it
> > special-purposed?
>
> Just as a specific example, what if my application indeed needs
> ExceptionGroup's (and not just more flexible matching of existing
> exception chains), but I want it to be a bit simpler: a) to have a flat
> structure of contained exceptions; b) want to actually emphasize the
> users that they should not rely on the ordering, so want it to be a set.
>
> My understanding is that ExceptionGroup is carefully designed to allow
> subclassing it and override behavior, and the above should be possible
> by subclassing.
Indeed:
class PaulsExceptionGroup(ExceptionGroup):
def __new__(cls, message, excs):
for e in excs:
if isinstance(e, BaseExceptionGroup):
raise TypeError("me is a flat exception group")
return super().__new__(cls, message, excs)
@property
def exceptions(self):
return set(super().exceptions)
def derive(self, excs):
return PaulsExceptionGroup(self.message, excs)
eg1 = PaulsExceptionGroup("Paul's group", [ValueError(12),
TypeError(42)])
print(repr(eg1))
print(eg1.exceptions)
match, rest = eg1.split(ValueError)
print(f'{match=!r}\n{rest=!r}')
print(f'{match.exceptions=}\n{rest.exceptions=}')
Output:
PaulsExceptionGroup("Paul's group", [ValueError(12), TypeError(42)])
{TypeError(42), ValueError(12)}
match=PaulsExceptionGroup("Paul's group", [ValueError(12)])
rest=PaulsExceptionGroup("Paul's group", [TypeError(42)])
match.exceptions={ValueError(12)}
rest.exceptions={TypeError(42)}
See - split() just works for you out of the box.
But usually, a simpler base class is subclassed to have
> more advanced/complex behavior. So again, current ExceptionGroup feels
> kind of specialized with its careful attention to hierarchy and
> multi-story "3D" tracebacks.
So your concern with our design is that ExceptionGroup implements a generic
split() that handles tracebacks and nested structure correctly, and you
might not need that because maybe you don't nest or you don't care about
tracebacks?
If that is the case then I think you are confusing "generic" with
"special-purpose". ExceptionGroup is generic, it works for general nested
groups with arbitrary tracebacks and cause/context. That's the opposite of
special-purpose.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SKIJ7ZG6AF2UFXBCGYAODX4F6M3GSPPP/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
Hi Paul,
IIUC, you are saying that exception group should not be a builtin type
because it is (1) complex (2) special-purposed. Instead, you propose that
we make exception handling pluggable.
(1) I agree, it is somewhat complex - it took us several iterations to get
from the idea of "a container of exceptions" to something that works
correctly with split()/subgroup(), maintaining the integrity of
__traceback__/__cause__/__context__ information, allowing subclassing and
having a relatively simple API. To me this is a reason to provide it as a
builtin rather than a reason not to (I think we should add this point to
the PEP- thanks).
You wrote:
> So yes, maybe being well aware that you're handling exactly Trio's (or
asyncio's, or context manager's) error isn't bad thing actually.
Maybe, in which case you can create a subclass of ExceptionGroup to mark
the groups coming from your library. But also maybe not, in which case you
don't have to.
(2) special-purposed? How so? When I asked for a practical advantage of
your pluggable solution I meant an example of a use case that our proposal
doesn't accommodate well and where a pluggable one does better.
The only example you suggested so far was the TemporaryDirectory one, which
I don't find compelling:
> > def chained(e, excs):
> > while e:
> > if isinstance(e, excs):
> > return e
> > e = e.__cause__ # Simplified, should consider __context__
> > too
> >
> > try:
> > tempfile.TemporaryDirectory(...)
> > except *chained(OSError) as e:
> > print(e)
What if the user code raised an OSError too, so now that exception group
has more than one? Even without that problem, why should I prefer this to
our proposal?
I am interested to know (1) what limitations you see in the ExceptionGroup
type we are proposing (in what sense is it special-purposed? what purpose
is it unsuitable for?) and (2) an example where the pluggable solution
works better.
Irit
On Sat, Mar 27, 2021 at 12:42 PM Paul Sokolovsky wrote:
> Hello,
>
> On Sat, 27 Mar 2021 10:55:40 +
> Irit Katriel wrote:
>
> > One of the motivations for introducing ExceptionGroup as a builtin is
> > so that we won't have a different custom version in each library that
> > needs it. So if you are writing a library the needs to raise multiple
> > exceptions, and then you decide to call Trio, you don't need to
> > translate Trio's MultiError into your own exception group type,
> > because everybody uses the builtin.
>
> Looking from a different angle shows a different perspective:
>
> 1. Trio devised an interesting concept of "nurseries" to deal with
> multiple tasks in asyncio-like programming.
>
> 2. It was all nice and beautiful until ... it came to error handling.
> Note that it's a rather typical situation - one can write nice, clean,
> beautiful code, which is not adequate in real-world scenarios,
> particularly because of error handling concerns. Oftentimes, such
> initial code/concepts are discarded, and more robust (though maybe not
> as beautiful) concepts/code is used.
>
> 3. But that's not what happened in the Trio case. The concept of
> nurseries got pushed forward, until it became clear that it requires
> changes on the programming language level.
>
> 4. That's how PEP654 was born, which, besides async scheduling of
> multiple tasks, bring *somewhat similar* usecases of e.g. raising
> exceptions thru context managers' own exceptions.
>
> Note that where errors and exceptions lead us is in questions on how to
> handle them. And beyond a couple of well known patterns ("dump and
> crash" or "dump and continue with next iteration"), error handling is
> very adhoc to a particular application and particular error(s). Seen
> from that angle, Trio wants to vendor-lock the entire language into its
> particular (experimental) way of handling multiple errors.
>
> So yes, maybe being well aware that you're handling exactly Trio's (or
> asyncio's, or context manager's) error isn't bad thing actually. And if
> it's clear that multiple asyncio frameworks are interested in sharing
> common exception base class for such usecases, then it could be
> introduced on the "asyncio" package level, or maybe even better, as a
> module similar to "concurrent.futures".
>
> > And your users don't need to
> > learn how your particular exception group works because they know
> > that you are using the builtin one.
> >
> > I see the aesthetic value of your suggestion, but does it have
> > practical advantages in light of the above?
>
> The concern is that it codifies pretty complex and special-purpose
> things on the language level. And it seems that the whole concept is
> rather experimental and "original design". We can compare that with
> another sufficiently complex feature which landed recently: pattern
> matching. At all phases of the design and discussion of that feature,
> one of the guiding principles was: "Many other languages implement it,
> so we know it's generally
[Python-Dev] Re: PEP 654: Exception Groups and except* [REPOST]
One of the motivations for introducing ExceptionGroup as a builtin is so that we won't have a different custom version in each library that needs it. So if you are writing a library the needs to raise multiple exceptions, and then you decide to call Trio, you don't need to translate Trio's MultiError into your own exception group type, because everybody uses the builtin. And your users don't need to learn how your particular exception group works because they know that you are using the builtin one. I see the aesthetic value of your suggestion, but does it have practical advantages in light of the above? Irit On Sat, Mar 27, 2021 at 10:31 AM Paul Sokolovsky wrote: > Hello, > > On Fri, 26 Mar 2021 16:19:26 -0700 > Guido van Rossum wrote: > > > Everyone, > > > > Given the resounding silence I'm inclined to submit this to the > > Steering Council. While I'm technically a co-author, Irit has done > > almost all the work, and she's done a great job. If there are no > > further issues I'll send this SC-wards on Monday. > > One issue with PEP654 is that it introduces pretty adhoc and > complex-semantics concept (ExceptionGroup) on the language level. > Here's an idea (maybe duplicate) on how to introduce a much simpler, > and more generic concept on the language level, and let particular > frameworks to introduce (and elaborate without further changing the > language) adhoc concept they need. > > So, let's look how the usual "except MyExc as e" works: it performs > "isinstance(e0, MyExc)" operation, where e0 is incoming exception > (roughly, sys.exc_info[1]), and if it returns True, then assigns e0 to > the "e" variable and executes handler body. "isinstance(e0, MyExc)" is > formally known as an "exception filter". > > As we see, currently Python hardcodes isinstance() as exception filter. > The idea is to allow to use an explicit exception filter. Let's reuse > the same "except *" syntax to specify it. Also, it's more flexible > instead of returning True/False from filter, to return either None > (filter didn't match), or an exception object to make available to > handler (which in general may be different than passed to the filter). > With this, ExceptionGroup usecases should be covered. > > Examples: > > 1. Current implicit exception filter is equivalent to: > > def implicit(e0, excs): # one or tuple, as usual > if isinstance(e0, excs): > return e0 > return None > > try: > 1/0 > except *implicit(ZeroDivisionError) as e: > print(e) > > 2. Allow to catch main or chained exception (context manager example > from PEP) > > def chained(e, excs): > while e: > if isinstance(e, excs): > return e > e = e.__cause__ # Simplified, should consider __context__ too > > try: > tempfile.TemporaryDirectory(...) > except *chained(OSError) as e: > print(e) > > 3. Rough example of ExceptionGroup functionality (now not a language > builtin, just implemented by framework(s) which need it, or as a > separate module): > > class ExceptionGroup: > > ... > > @staticmethod > def match(e0, excs): > cur, rest = e0.split_by_types(excs) > # That's how we allow an exception handler to re-raise either an > # original group in full or just "unhandled" exception in the > # group (or anything) - everything should be passed via > # exception attributes (or computed by methods). > cur.org = e0 > cur.rest = rest > return cur > > try: > ... > except *ExceptionGroup.match((TypeError, ValueError)) as e: > # try to handle a subgroup with (TypeError, ValueError) here > ... > # now reraise a subgroup with unhandled exceptions from the > # original group > raise e.rest > > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ZYMKRVYLUF7TWDWSCPXCRZFBDO3YOZMF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 654: Exception Groups and except* [REPOST]
We would like to present for feedback a new version of PEP 654, which incorporates the feedback we received in the discussions so far: https://www.python.org/dev/peps/pep-0654/ The reference implementation has also been updated along with the PEP. The changes we made since the first post are: 1. Instead of ExceptionGroup(BaseException), we will have two new builtin types: BaseExceptionGroup(BaseException) and ExceptionGroup(BaseExceptionGroup, Exception). This is so that "except Exception" catches ExceptionGroups (but not BaseExceptionGroups). BaseExceptionGroup.__new__ inspects the wrapped exceptions, and if they are all Exception subclasses, it creates an ExceptionGroup instead of a BaseExceptionGroup. 2. The exception group classes are not final - they can be subclassed and split()/subgroup() work correctly if the subclass overrides the derive() instance method as described here: https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups 3. We had some good suggestions on formatting exception groups, which we have implemented as you can see in the output shown for the examples in the PEP. 4. We expanded the section on handling Exception Groups, to show how subgroup can be used (with side effects) to do something for each leaf exception, and how to iterate correctly when the tracebacks of leaf exceptions are needed: https://www.python.org/dev/peps/pep-0654/#handling-exception-groups 5. We expanded the sections on rationale and backwards compatibility to explain our premise and expectations regarding how exception groups will be used and how the transition to using them will be managed. 6. We added several items to the rejected ideas section. We did not receive any comments (or make any changes) to the proposed semantics of except*, hopefully this is because everyone thought they are sensible. Irit, Yury and Guido ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MQ2UCSQ2ZC4FIGT7KSVI6BJA4FCXSOCL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
We believe that we have found a simple way to make it possible to subclass exception groups, with fully functioning split() and subgroup(). See this section in the PEP: https://www.python.org/dev/peps/pep-0654/#subclassing-exception-groups It was also added to the reference implementation. This probably opens new possibilities for adoption strategies of exception groups and except* (in terms of the interaction of exception groups with except). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/33AQ3V3RXRBWWCILJNCEIEERNGAG3M2T/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Mar 4, 2021 at 11:15 PM Glenn Linderman wrote: > I like explicit, and avoiding magic. > > And this gives a compatibility story for outer loops that except: > Exception, and even for others cases that are not recoded for > ExceptionGroup handling. > It could help during the migration/mixed python version phase. But I'm not convinced yet that this is what we want to end up with in the long term. Waiting to hear more thoughts. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VITKAEOCPYMWN6EOZAGT23A3Y2ATOHC4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Mar 4, 2021 at 8:48 PM Sven R. Kunze wrote: > Hi Irit, > > makes sense. So, in case of a *mixed-type ExceptionGroup,* SystemExit > wins and forces the program to exit. > > > Could you add your reasoning to the PEP? > Good idea, I'll add "ExceptionGroup(BaseException)" as a rejected idea and explain it there. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WJUKFBFDVRW5VQK74QKZVYFSAQO3FLBT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Whether you have "as" or not, the value of sys.exc_info() (which is what would be attached as the context to anything you raise in the except block) is the same. So these are not two different cases -- the only difference is whether or not you have a local variable set to sys.exc_info(). On Thu, Mar 4, 2021 at 4:46 PM Baptiste Carvello < devel2...@baptiste-carvello.net> wrote: > Hi, > > I'll take a shoot at this, just to see how it tastes… So, let's say: > > When an exception group reaches a set of traditional "except" clauses, > those are examined one after the other, in the order they are in the > code. That way, exceptions matched by several clauses will cause the > first one to run, same as today. > > A subgroup is built with the subset of exceptions matched by the > examined clause, as the PEP specifies for "except*". If this subgroup is > None, the clause is not selected, and the next clause, if any, is > examined. On the contrary, if the subgroup contains at least one matched > exception, the clause is selected and no other clause will run (again, > same as today). Exceptions not part of the subgroup are discarded. > > The clause body is then run just once (so the boss only gets one email > about KeyboardInterrupt). If the clause uses the "as" form, the "as" > variable is bound to one exception in the subgroup, which one is > unspecified (at least for now). The other ones are discarded, except if > a bare "raise" is reached (below). > > If a bare "raise" is reached while executing the body, the selected > subgroup propagates out of the "try-except" construct. Justification: > the whole group cannot propagate, because today a bare "raise" cannot > reraise exceptions of a type not matched by the clause. However, if a > single-type exception group is handled similar to a single exception in > traditional "except" clauses, it is acceptable to let it propagate. > > So you would have: > > try: > g=BaseExceptionGroup( > [ValueError(), KeyboardInterrupt(), KeyboardInterrupt()]) > raise g > except RuntimeError: # doesn't match > log_the_error() > except KeyboardInterrupt as e: # builds s=g.subgroup(KeyboardInterrupt) > email_the_boss(e) # tells the boss of any one error > raise # reraises s > except BaseException: # would match, but doesn't run > launch_nuclear_attack() > > # BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt()]) > # propagates further, a traditional "except KeyboardInterrupt" > # would catch it. The ValueError is discarded. > > An interesting feature would be: when the matching clause has no "as", > "except" behaves the same as "except*", apart from the fact that only > one clause may run. > > Cheers, > Baptiste > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/EXC6BQVHQXUXBHIEWHLSLU6FY7SJKIF3/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TN2DVSAATJSYJ6EVWVRK2Z4EBXWAYUCT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Mar 4, 2021 at 1:38 AM Glenn Linderman
wrote:
> On 3/3/2021 2:49 PM, Irit Katriel via Python-Dev wrote:
>
> That's an interesting idea.
>
> Do you mean that one exception gets handled and the rest of the group is
> reraised? Or discarded?
>
> The value of sys.exc_info() (and the e in "except T as e:") needs to be a
> single naked exception. So if there is more than one match in the group we
> would need to pick one (let's say the first in DFS order).
>
> If we do this, then we have this situation. Before ExceptionGroups, you
> got to choose which of the exceptions you have is the most important, and
> you raised only that one. Now you raise a bunch of them and the order of
> the except clauses in caller's code determines which one of them counts and
> which ones are discarded. What do you make of that?
>
>
> You _could_ implement it as you said, but remember, you that with this
> idea, you are changing how except clauses work—so instead of making the
> order of the except clauses determine which one counts most, you could
> instead do something else.
>
> One alternative idea would be to take the "first in DFS order" and see if
> it matches any of the except clauses, and if so, process that one. If not,
> then pick the next, and see if it matches, until one is found that matches,
> and can be processed.
>
Or we could make it explicit:
add an optional arg to ExceptionGroup like
ExceptionGroup("eg", list_of_exceptions, singleton=None)
In the example of atexit, where currently it raises only the last exception
from your callbacks, it will instead raise
ExceptionGroup("atexit errors", all_exceptions, singleton=last_exception)
Then except* works as before, ignoring the singleton. But except matches
the singleton.
And there's no magic where you can be surprised about which exception
except chose to look at.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/LWIDPI5ZTXAPO4ATKBYRVZPUWHWVLSNT/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Wed, Mar 3, 2021 at 10:39 PM Greg Ewing wrote: > On 4/03/21 5:37 am, Paul Moore wrote: > > frameworks and libraries typically have to interact with other users' > > code, and there the contract has changed from "do what you like in > > your code and I'll cope" to "do what you like in your code as long as > > you don't let an exception group escape, and I'll cope"... And I have > > to change *my* code to get the old contract back. > > Seems to me this whole issue would go away if the ordinary > except statement were to look inside ExceptionGroups. > > In other words, the only difference between except and > except* would be that multiple except* clauses can be run, > whereas only one except clause will run (the first one that > matches something in the ExceptionGroup). > > Is there any good reason not to do things that way? > That's an interesting idea. Do you mean that one exception gets handled and the rest of the group is reraised? Or discarded? The value of sys.exc_info() (and the e in "except T as e:") needs to be a single naked exception. So if there is more than one match in the group we would need to pick one (let's say the first in DFS order). If we do this, then we have this situation. Before ExceptionGroups, you got to choose which of the exceptions you have is the most important, and you raised only that one. Now you raise a bunch of them and the order of the except clauses in caller's code determines which one of them counts and which ones are discarded. What do you make of that? Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O72A2J4HRHEKYBRL5NRL4YEUQVSL3TZQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Sven, This is all about the popularity of "except Exception". Look at the hierarchy of builtin exceptions: https://docs.python.org/3/library/exceptions.html#exception-hierarchy Most exceptions are subclasses of Exception. There are a few which are not, because they typically mean "this process should exit" so you should not usually handle them in your code. People use "except Exception" as a way to "catch almost everything, but not the critical stuff like SystemExit". If we make ExceptionGroup be a BaseException, then "except Exception" doesn't catch it. So we make it a subclass of Exception. But then we can't make it wrap things like SystemExit, which people expect will not be caught by "except Exception". So we add BaseExceptionGroup, which is a subclass of BaseException and therefore is not caught by "except Exception", so it can wrap SystemExit. Why is the choice automated? Because it can be. You look at what you're wrapping. If it's all subclasses of Exception, then it can be ExceptionGroup. If there are BaseExceptions, then it needs to be BaseExceptionGroup. There is no reason to ever do anything else. I hope that makes sense. On Wed, Mar 3, 2021 at 7:32 PM Sven R. Kunze wrote: > Hey Irit, > > find my 3 answers below: > > On 03.03.21 13:17, Irit Katriel wrote: > > Hi Sven, > > > > I like your formatting suggestion, thanks. I will do something like that. > > You're welcome. > > > > > I'm not sure I understand your question. ExceptionGroup is a subclass > > of Exception (which is a subclass of BaseException). So ExceptionGroup > > is caught by "except Exception" or "except BaseException". > > 1) So I understand "try-except BaseException" (cf. concurrent.futures) > will work without fixes (i.e. produce the same results). > > > > BaseExceptionGroup is a subclass only of BaseException so it is caught > > by "except BaseException" but not "except Exception". And > > ExceptionGroup is allowed to wrap only Exceptions while BaseException > > can wrap Exceptions and and BaseExceptions. Makes sense? > > > 2) Can you add motivating examples for "BaseExceptionGroup vs > ExceptionGroup" in the PEP? Right now, I only see what the consequences > are but not why it was done this way. > > 3) Can you explain (and show the reasoning behind) this automatic choice > in the PEP? Sounds a bit like hidden magic to me. > > > Referring to: "The difference between them is that ExceptionGroup can > only wrap Exception subclasses while BaseExceptionGroup can wrap any > BaseException subclass. A factory method that inspects the nested > exceptions and selects between ExceptionGroup and BaseExceptionGroup > makes the choice automatic." > > > Best > Sven > > > PS: > > the reason why I was a bit puzzled by the > BaseExceptionGroup/ExceptionGroup issue is that: > - if it doesn't matter (so we do it automatically, because we do not > want to bother anybody), why do we need ExceptionGroup at all, > BaseExceptionGroup seems more flexible? > - if it does matter, why is the choice automatic and what if it was the > wrong choice? > > > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JZJFDTZKPIY477EAVWHWEHLX2XFEAXJK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Wed, Mar 3, 2021 at 6:57 PM Paul Moore wrote: > Sorry, I keep thinking I've finished and you keep making interesting > points :-) > > On Wed, 3 Mar 2021 at 17:01, Irit Katriel > wrote: > > > Raising an ExceptionGroup is an API change. If you call APIs that say > they will raise ExceptionGroups you need to update your code accordingly. > If a library doesn't document that it raises ExceptionGroups and then one > of those escapes, then that library has a bug. Just like with any other > exception type. > > In my experience, libraries don't document what exceptions they raise > very well. You can call that a bug, but it's a fact of life, I'm > afraid. The problem here isn't so much that the library code now > raises an exception that it used not to raise, but rather that *the > user hitting Ctrl-C* can now result in a different exception surfacing > in my code than it used to. Libraries don't re-wrap KeyboardInterrupt, > as you pointed out in a previous response, so I can currently write > code that traps KeyboardInterrupt, safe in the knowledge that by doing > so I'll handle that user action properly. But with PEP 654, libraries > might well (indeed, some libraries almost certainly will) start > wrapping KeyboardInterrupt in an exception group. That's a backward > incompatible change from the perspective of my code's interaction with > the user, and I need to re-code my application to deal with it (and > worse still, writing that new code in a way that is portable between > versions is not particularly straightforward). > This is also true for MemoryError, and many other errors. What makes KeyboardInterrupt special? > > > For older Pythons you would have to do something like > > > > except KeyboardInterrupt: > >... > > except BaseExceptionGroup: # some stub type in old versions > ># inspect the contents > ># if there's a KeyboardInterrupt do what you need to do > ># reraise the rest > > I'd be inclined to suggest that a complete version of this should be > included in the "Backward compatibility" part of the PEP, as I > honestly don't really know how I'd write that without doing more > research. But such an example would make the KeyboardInterrupt case > seem more important than it is. Maybe if it's framed as "how to write > calling code that's compatible with older versions of Python but still > able to handle called code potentially raising exceptions that you > need to trap as part of a group", that would be a useful general > example. > Or maybe it's not actually something that will be needed that often. > I'm not sure - I'm trying to think in terms of pip, where we can't use > new features in our own code until we drop support for older versions, > but we might potentially rely on a library that uses exception > grouping internally on versions where it's available (and that code > lets those exception groups escape). It feels like a stretch to claim > this is particularly likely, but conversely it's something I can > easily imagine *could* happen... If a library starts raising ExceptionGroups from version 3.X it should probably do that from a new API so people won't have to worry about it just because they are bumping Python version. So I think the cross-version issue is in the case of "I'm calling a user function and I don't know what it is or what it raises", or the "I just want to write all exceptions to the log and ignore them". So this is the "except Exception" case, which will work. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BF6R7Y6UAPSAFZYBQ2GKAMO7RJ6FYDAY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Wed, Mar 3, 2021 at 4:37 PM Paul Moore wrote:
>
> Similar to the argument for "except Exception". Applications that trap
> KeyboardInterrupt so that they can exit cleanly without an ugly
> traceback will no longer trap *all* keyboard interrupts, as they could
> miss grouped ones.
>
See below.
>
> If we accept that grouped exceptions should never escape out of a
> well-defined context, then this wouldn't be such an issue. But there's
> nothing in the PEP that enforces that, and there *is* code that needs
> to be prepared for "any sort of result". It's the except Exception
> argument again.
>
> So code that wants to exit cleanly in the face of Ctrl-C will need to
> be rewritten from:
>
> try:
> main()
> except KeyboardInterrupt:
> print("User interrupted the program. Exiting")
> sys.exit(1)
>
> to:
>
> try:
> try:
> main()
> except KeyboardInterrupt:
> print("User interrupted the program. Exiting")
> sys.exit(1)
> except *KeyboardInterrupt:
> print("User interrupted the program. Exiting")
> sys.exit(1)
>
It suffices to do
try:
main()
except *KeyboardInterrupt:
print("User interrupted the program. Exiting")
sys.exit(1)
because "except *T" catches Ts as well.
>
> Did I miss an easier way of writing this code? And worse, how would I
> write it so that it was portable between Python 3.9 and later versions
> (which is a common requirement for library code - admittedly library
> code wouldn't normally be doing this sort of top-level trap, but it
> could just as easily be "catch Ctrl-C and do a bit of tidy-up and
> re-raise").
>
For older Pythons you would have to do something like
except KeyboardInterrupt:
...
except BaseExceptionGroup: # some stub type in old versions
# inspect the contents
# if there's a KeyboardInterrupt do what you need to do
# reraise the rest
>
> > I think the only reason you're comfortable with having to select between
> the exceptions that were raised and discard some of them is because that's
> where we are today. The PEP lists several standard library and other APIs
> that discard errors because they need to pick one. That's what we're trying
> to fix.
>
> Maybe. But I'm not looking at it as being "comfortable" with the
> current situation, but rather as "I don't use any of these new
> features, why am I having to change my code to accommodate stuff I
> don't use?" If I own the full stack, that's not an issue, but
> frameworks and libraries typically have to interact with other users'
> code, and there the contract has changed from "do what you like in
> your code and I'll cope" to "do what you like in your code as long as
> you don't let an exception group escape, and I'll cope"... And I have
> to change *my* code to get the old contract back.
>
"except Exception"/"except BaseException" is the special case of "I don't
know what I'm calling and I want to catch everything". And that (to
repeat, just in case) will work as you expect.
If you are actually handling exceptions selectively, then I can break you
already in 3.9 just by raising a different exception type to the one you
are catching. How is this different?
Raising an ExceptionGroup is an API change. If you call APIs that say they
will raise ExceptionGroups you need to update your code accordingly. If a
library doesn't document that it raises ExceptionGroups and then one of
those escapes, then that library has a bug. Just like with any other
exception type.
>
> But it's a small point in the wider scheme of things, and I'm not
> going to labour the point any more. Thanks for listening and taking
> the time to reply.
It's an important one though. Thanks for asking good questions.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SDHZAEUNHMOC2NBCFQZEDTHZ4B6QGEXO/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Paul, On Wed, Mar 3, 2021 at 2:20 PM Paul Moore wrote: > > 1. Having now read the PEP, I don't actually see a use case for > grouping BaseExceptions. Why would anyone catch KeyboardInterrupt or > SystemExit and wrap them in a BaseExceptionGroup anyway? It seems to > me that the right thing to do, even in async or parallel code, is to > just propogate the KeyboardInterrupt/SystemExit up to the main > program. Losing a ValueError that happened at the exact same time as > the user pressed Ctrl-C seems like it's not going to be a problem in > practice... > It is possible that your program wants to do something other than just terminate when it gets a KeyboardInterrupt. Otherwise why does the interpreter bother propagating the KeyboardInterrupt rather than just terminate the program there and then? And the ValueError you lost may be some other error type that involves cleanup. If you do this: try: async.gather(...) # raises BaseExceptionGroup( DoCleanup(), KeyboardInterrupt()) except * DoCleanup: do_cleanup() # log stuff, send notifications, close sockets, whatever That will do the cleanup in addition to propagating a BaseExceptionGroup(KeyboardInterrupt()) Also, KeyboardInterrupt/SystemExit are not the only BaseExceptions. > 2. Given the above, why even have a means of grouping BaseExceptions > at all? Why not just have ExceptionGroup that can only catch instances > of Exception? > Because the IGotInterrupted alternative involves wrapping a BaseException by an Exception, which is not something we should push people into doing (it's not that different from allowing ExceptionGroup to wrap BaseExceptions directly). What's the harm/complication in offering a BaseExceptionGroup(BaseException) in addition to ExceptionGroup(BaseExceptionGroup, Exception)? I think the only reason you're comfortable with having to select between the exceptions that were raised and discard some of them is because that's where we are today. The PEP lists several standard library and other APIs that discard errors because they need to pick one. That's what we're trying to fix. Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HHNSKCF4DEBAWR2PUAVQ2QXNSJMWMHS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Paul, I agree that your condition (1) is essential, while condition (2) is desirable. (I explained in the email you replied to why I think 2 is less critical than 1). The current state of the PEP is that ExceptionGroup does not wrap BaseExceptions and is caught by "except Exception", while BaseExceptionGroup wraps BaseException and is only caught by "except BaseException" but not "except Exception". This is covered the PEP, but TL;DR: If we could make "except KeyboardInterrupt" catch BaseExceptionGroup([KeyboardInterrupt]) in a reasonably backwards compatible way then we wouldn't need except*. For example, suppose that this: try: raise BaseExceptionGroup([KeyboardInterrupt()]) except KeyboardInterrupt: email_the_boss() worked as you suggest. Then what should this do? Send two emails to the boss? try: raise BaseExceptionGroup([KeyboardInterrupt(), KeyboardInterrupt() ]) except KeyboardInterrupt: email_the_boss() As you noted, no library is under any obligation to wrap KeyboardInterrupts into the exception groups it raises. You may decide it's a bad idea and not do it. What we are discussing here is what the language should make possible. We agree that wrapping a BaseException by an Exception is something we should definitely block. When it's wrapping a BaseException by another, new BaseException type, in my view that's ok. You may have a bug where you don't catch an exception you want to catch, because you are using a new API incorrectly. But you won't have bugs where you swallow an exception that you didn't swallow before. Irit On Wed, Mar 3, 2021 at 8:30 AM Paul Moore wrote: > On Tue, 2 Mar 2021 at 21:46, Irit Katriel via Python-Dev > wrote: > > As an aside - I found it interesting that the option to wrap > BaseException instances by an Exception, which came up a couple of times in > this thread, didn't generate any anxiety. > > Probably because it wasn't clear that was ever being proposed... (or > at least the implication wasn't obvious - presumably this is somehow > related to BaseExceptions being accumulated in ExceptionGroups?) :-( > > I would consider it essential that if someone hits Ctrl-C and that > generates a KeyboardInterrupt, then: > > 1. That KeyboardInterrupt will *not* get caught by exception handlers > only interested in Exception instances > 2. That KeyboardInterrupt *will* get caught by any handler that does > an explicit `except KeyboardInterrupt` or an `except BaseException`. > > To me, that's pretty much essential to correct Ctrl-C handling in any > app (never ignore a user's Ctrl-C and always exit cleanly if one is > raised). > > That might mean that BaseException instances shouldn't be "groupable", > but I don't want to comment on that until I've properly read the PEP > (I've skimmed it now, but only superficially). At a minimum, I'd > consider it a bug for library code to manually wrap a > KeyboardInterrupt in an exception group (just like code that catches > KeyboardInterrupt and re-raises it as a ValueError would be today). > > Paul > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5M6YSXXZWGUHE5VOFLJWKRSUY4TIXLTH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Sven, I like your formatting suggestion, thanks. I will do something like that. I'm not sure I understand your question. ExceptionGroup is a subclass of Exception (which is a subclass of BaseException). So ExceptionGroup is caught by "except Exception" or "except BaseException". BaseExceptionGroup is a subclass only of BaseException so it is caught by "except BaseException" but not "except Exception". And ExceptionGroup is allowed to wrap only Exceptions while BaseException can wrap Exceptions and and BaseExceptions. Makes sense? Irit On Tue, Mar 2, 2021 at 7:25 PM Sven R. Kunze wrote: > Just to be on the safe side here, I would want to asked a little bit > further. > > > On 02.03.21 12:22, Irit Katriel wrote: > > > 1) What is the right "except pattern" when ExceptionGroup is introduced >> for the use cases described above (remote or concurrent python processes)? >> > > If you want to catch the whole ExceptionGroup and format its traceback, > then you can just do "except ExceptionGroup as eg" and then > traceback.print_exception(eg). > > The except* syntax is for when you want to handle only certain types of > exceptions from the group, selectively. > > Just to make sure, I understand this completely. > > > In order to make it more tangible here is an example from the stdlib: > > > https://github.com/python/cpython/blob/bf2e7e55d7306b1e2fce7dce767e8df5ff42cf1c/Lib/concurrent/futures/process.py#L215 > > As you can see, BaseException is caught multiple times as the only > except-clause. _sendback_result(...) then used to transfer the > return_val/exception back to parent process. > > > How is the supposed way of handling a raised ExceptionGroup? > > > a) will the existing code simply work as it? > b) if not what are the changes to this lib specifically as a blueprint > example for others > > > Reading from the other subthread for this PEP, I am not 100% sure now that > "except BaseException" will suffice if ExceptionGroup inherits from > Exception instead of BaseException because it seems that ExceptionGroup > somehow is handled specially. So, why I try to approach this very issue > with existing code. Once that is clear, it should be easy for everybody to > apply the same pattern to their code. Here is the copy from github: > > > try: > r = call_item.fn(*call_item.args, **call_item.kwargs) > except BaseException as e: > exc = _ExceptionWithTraceback(e, e.__traceback__) > _sendback_result(result_queue, call_item.work_id, exception=exc) > else: > _sendback_result(result_queue, call_item.work_id, result=r) > del r > > > Maybe even _sendback_result could benefit from using ExceptionGroup > itself. You can see there another "except BaseException" in case of an > error. But that's maybe a different topic. > > > >> 2) What is the recommended approach of printing the traceback potentially >> incorporating multiple tracebacks - I couldn't find it in the PEP and >> tracebacks are a really neat tool for error hunting. >> > > Part of the proposal of the PEP is that we teach the builtin traceback > formatting code to display ExceptionGroups with all that information. > > > As long as there's the possibility to receive the usual (two-line) > entry-based list, I guess that is enough for every one to work with it > (e.g. hiding unnecessary tb entries). > > > The reference implementation has this, and the examples in the PEP were > produced with it. Some of the examples (the early ones) show exceptions > that were never raised so there is no traceback. But if you scroll down to > the later examples, you see the output for exceptions with tracebacks, > cause, context etc. > > > Ah I see them now. Thank you. :-) > > > We didn't put too much effort into making the traceback output pretty, so > at the moment it's a draft. If there is an ascii artist out there who has > suggestions on improving this, that would be great. > > > Well, yes. It's not really pretty. I would recommend a tree-based solution > such as the following: > > > Traceback (most recent call last): > File "", line 3, in > ExceptionGroup > | > +---+ Traceback (most recent call last): > | | File "", line 3, in > | | ExceptionGroup: eg > | | | > | | +---+ ValueError: a > | | > | | During handling of the above exception, another exception occurred: > | | > | | Traceback (most recent call last): > | | File "", line 5, in > | | KeyError: 'x' > | > | > +---+ Traceback (most recent call last): > | | File "", line 3, in > | | ExceptionGroup: eg > | | | > | | +---+ TypeError: b > > > I used a similar pattern for the remote-execution lib and received good > user feedback (even requesting color encoding for each layer of the tree > (not the text), so people know what belongs together). Besides colors, I > used https://en.wikipedia.org/wiki/Box-drawing_character but I guess > pipes, dashes and pluses could suffice for CPython. > > > One other remark from my side here: in the
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Barry, On Mon, Mar 1, 2021 at 6:12 PM Barry Scott wrote: > > Have you considered alternative syntax to the "except*"? > I feel that expect* is too easy to confuse with plain except. > We briefly considered things like "except Group[ValueError, TypeError]". Do you (or anyone else) have any suggestions on syntax? > > As others have said needing to change all except Exception: > handlers would be a breaking change for code that I maintain. > > I think its accepted that that idiom should continue to be as > reliable as it currently is. Can you confirm that I have understood > the conversation on this point? > That's not happening. The PEP has been updated regarding this. It was never going to be all "except Exception", most of them would not be affected. But we have a better plan now so this discussion is moot. As an aside - I found it interesting that the option to wrap BaseException instances by an Exception, which came up a couple of times in this thread, didn't generate any anxiety. In my day job I maintain a proprietary DSL that ~4000 developers use, and my experience has been that a bug that swallows an exception you need is worse than a bug that makes you get an exception you did not expect. The latter can hurt you once (and then you fix it). But the former is (1) harder to detect (2) longer to debug (3) once you know about it there is nothing you can do to work around it. Problems like that cost us years and years of work. > > I have been thinking about why I would be getting an ExceptionGroup > with, for example, a number of OSError for a handler to deal with. > Would it not be a difficult piece of code to write without the context of > each OSError? > > What I'm think is that the lower level pieces would be catching the OSError > closer to the source of the problem and raising a App/Lib specific > exception > that provides the context. For example AppFailedToSaveConfigError and > AppFailedToSaveDataError as oppose to a two permission OSError's. > I think you're right about this. OSError is an example of an exception with a data field that is often inspected for its value, so we picked on it to talk about how you would do something like this. Most of the time this won't be needed, but we still need to know that it's possible and reasonably ergonomic to do it. > With context I can imagine that handling the ExceptionGroup would be > a lot easier for the App designer. > > If that the pattern that emerges then is the complexity of nested > ExceptionGroups going to only rarely used? > I can't imagine people building deep trees of exceptions in practice (at least not on purpose). But some nesting may show up naturally, and we need to support it because otherwise it can get awkward if, for example, asyncio.gather() needs to wrap an exception group that came from a TemporaryDirectory.__exit__(). Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3W6C3Y6WVKGYL4H63VDLB7VF6JH52TV4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Sven, On Mon, Mar 1, 2021 at 8:59 PM Sven R. Kunze wrote: > Hey Irit, > > cool proposal. > Thank you. > I just have two questions regarding "except Exception" or "except > BaseException" as it is used e.g. by concurrent.futures.process (def > _process_worker) from the stdlib. > > Almost similarly, I maintain a library using this pattern to wrap/unwrap > exceptions from remote Python processes to create nicely formatted > tracebacks (also recursively of course if needed) at the calling python > process. > > Usually these exceptions are wrapped, transferred to the parent process, > there is the current call stack added, then reraised as a different > exception, and so on and so forth if a chain of parents exist. The > outermost parent process takes care of printing the tb. > > > My two questions regarding PEP 654: > > > 1) What is the right "except pattern" when ExceptionGroup is introduced > for the use cases described above (remote or concurrent python processes)? > If you want to catch the whole ExceptionGroup and format its traceback, then you can just do "except ExceptionGroup as eg" and then traceback.print_exception(eg). The except* syntax is for when you want to handle only certain types of exceptions from the group, selectively. > 2) What is the recommended approach of printing the traceback potentially > incorporating multiple tracebacks - I couldn't find it in the PEP and > tracebacks are a really neat tool for error hunting. > Part of the proposal of the PEP is that we teach the builtin traceback formatting code to display ExceptionGroups with all that information. The reference implementation has this, and the examples in the PEP were produced with it. Some of the examples (the early ones) show exceptions that were never raised so there is no traceback. But if you scroll down to the later examples, you see the output for exceptions with tracebacks, cause, context etc. We didn't put too much effort into making the traceback output pretty, so at the moment it's a draft. If there is an ascii artist out there who has suggestions on improving this, that would be great. Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N6ACTVYIPUXQGUXF7EGFXTX27VAA45L3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Cameron,
If you go long, I go longer :)
On Sun, Feb 28, 2021 at 10:51 PM Cameron Simpson wrote:
> On 28Feb2021 10:40, Irit Katriel wrote:
> >split() and subgroup() take care to preserve the correct metadata on
> >all
> >the internal nodes, and if you just use them you only make safe
> operations.
> >This is why I am hesitating to add iteration utilities to the API. Like we
> >did, people will naturally try that first, and it's not the safest API.
>
> Wouldn't it be safe if the ExceptionGroup were immutable, as you plan?
> Or have you another hazard in mind?
>
Making them immutable won't help the metadata issue. split() and subgroup()
copy the (context, cause traceback) from the original ExceptionGroups (root
and internal nodes of the tree) to the result trees. If you DIY creating
new
ExceptionGroups you need to take care of that.
> But all that said, being able to iterable the subexceptions seems a
> natural way to manage that:
>
> unhandled = []
> try:
> .
> except *OSError as eg:
> for e in eg:
> if an ok exception:
> handle it
> else:
> unhandled.append(e)
> if unhandled:
> raise ExceptionGroup("unhandled", unhandled)
>
You just lost the metadata of eg. It has no context, cause and its
traceback begins here. And the exceptions contained in it, if they came
from a deeper tree that you flattened into the list, now look like their
traceback jumps straight to here from the place they were actually first
inserted into an ExceptionGroup. This may well be an impossible code path.
Here's an example:
>>> import traceback
>>>
>>> def flatten(exc):
... if isinstance(exc, ExceptionGroup):
... for e in exc.errors:
...yield from flatten(e)
... else:
... yield exc
...
>>> def f():
... try:
... raise ValueError(42)
... except ValueError as e:
... return e
...
>>> def g():
... try:
... raise ExceptionGroup("g", [f()])
... except ExceptionGroup as e:
... return e
...
>>> def h():
... try:
... raise ExceptionGroup("h", [g()])
... except ExceptionGroup as e:
... return e
...
>>> def flat_h():
... try:
... raise ExceptionGroup("flat_h", list(flatten(h(
... except ExceptionGroup as e:
... return e
...
>>>
>>> traceback.print_exception(h())
Traceback (most recent call last):
File "", line 3, in h
ExceptionGroup: h
Traceback (most recent call last):
File "", line 3, in g
ExceptionGroup: g
Traceback (most recent call last):
File "", line 3, in f
ValueError: 42
>>> traceback.print_exception(flat_h())
Traceback (most recent call last):
File "", line 3, in flat_h
ExceptionGroup: flat_h
Traceback (most recent call last):
File "", line 3, in f
ValueError: 42
>>>
traceback.print_exception(h()) prints a reasonable traceback - h() called
g() called f().
But according to traceback.print_exception(flat_h()), flat_h() called
f().
You can preserve the metadata (and the nested structure with all its
metadata) if you replace the last line with:
raise eg.subgroup(lambda e: e in unhandled)
And for the part before that, iteration, Guido's pattern showed that you
can roll it into the subgroup callback.
>
> There are some immediate shortcomings above. In particular, I have no
> way of referencing the original ExceptionGroup without surprisingly
> cumbersome:
> try:
> .
> except ExceptionGroup as eg0:
> unhandled = []
> eg, os_eg = eg0.split(OSError)
> if os_eg:
> for e in os_eg:
> if an ok exception:
> handle it
> else:
> unhandled.append(e)
> if eg:
> eg, value_eg = eg.split(ValueError)
> if value_eg:
> for e in value_eg:
> if some_ValueError_we_understand:
> handle it
> else:
> unhandled.append(e)
> if eg:
> unhandled.append(eg)
> if unhandled:
> raise ExceptionGroup("unhandled", unhandled) from eg0
>
This is where except* can help:
try:
.
except except *OSError as eg:
unhandled = []
handled, unhandled = eg.split(lambda e: e is an ok exception) #
with side effect to handle e
if unhandled:
raise unhandled
except *ValueError as eg:
handled, unhandled = eg.split(lambda e: e is a value error we
understand) # with side effect to handle e
if unhandled:
I have the following concerns with the pattern above:
>
> There's no way to make a _new_ ExceptionGroup with
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Sun, Feb 28, 2021 at 6:17 PM Guido van Rossum wrote: > On Sun, Feb 28, 2021 at 2:40 AM Irit Katriel > wrote: > >> >> In earlier versions of the PEP ExceptionGroup was not immutable and split >> actually removed matching exceptions from it. >> It was also iterable so you could get a flat list of all the contained >> leaf exceptions. Then we changed it. >> >> ExceptionGroup is a tree of exceptions, and the internal nodes of the >> tree (which are ExceptionGroups) have metadata on them - context, cause, >> traceback. >> If you want the full traceback of a leaf exception you need to >> concatenate the tracebacks of all the exceptions on the path from the root >> to the leaf. >> If you flatten an ExceptionGroup and create a new list >> ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has >> depth 1). >> > > Is this a typo? Did you mean "If you flatten [it] and create a new list... > you *do* lose metadata (unless ... depth 1)"? > It is a typo, thanks. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/M3DF5VIJWVE73SU4576ICKOGLEEWQBNO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
In earlier versions of the PEP ExceptionGroup was not immutable and split actually removed matching exceptions from it. It was also iterable so you could get a flat list of all the contained leaf exceptions. Then we changed it. ExceptionGroup is a tree of exceptions, and the internal nodes of the tree (which are ExceptionGroups) have metadata on them - context, cause, traceback. If you want the full traceback of a leaf exception you need to concatenate the tracebacks of all the exceptions on the path from the root to the leaf. If you flatten an ExceptionGroup and create a new list ExceptionGroup(list(eg)) you don't lose metadata (unless eg's tree has depth 1). Similarly, if you add an exception it needs to have matching metadata to the ExceptionGroup you are adding it to. It's too easy to get that wrong. split() and subgroup() take care to preserve the correct metadata on all the internal nodes, and if you just use them you only make safe operations. This is why I am hesitating to add iteration utilities to the API. Like we did, people will naturally try that first, and it's not the safest API. We actually have the OSErrors example in the PEP, just above https://www.python.org/dev/peps/pep-0654/#caught-exception-objects: try: low_level_os_operation() except *OSerror as errors: raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None On Sun, Feb 28, 2021 at 6:30 AM Guido van Rossum wrote: > We really don’t want users pushing non-exceptions into the list, nor do we > want e.g. KeyboardInterrupt to be added to a (non-Base-) ExceptionGroup. > > There’s a pattern for what you propose using the split() method and a > lambda, or you can keep the exceptions in a list and re-wrap them at the > end. > > On Sat, Feb 27, 2021 at 20:41 Cameron Simpson wrote: > >> On 26Feb2021 02:44, Irit Katriel wrote: >> >On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum >> wrote: >> >> OT: Is ExceptionGroup *really* immutable in the current >> >> implementation? As >> >> long as the 'errors' field is a list, I think one could mutate the list >> >> directly. >> > >> >It's not, but we were going to make it an immutable tuple. >> >> Could you say why? Other than wanting to discourage mutation happy code >> getting out there? >> >> The reason I ask is that the scenario which comes to my mind is >> something like: >> >> except *OSError as e: >> >> AIUI "e" is an ExceptionGroup containing only OSErrors. So, one common >> thing in my own code is this: >> >> try: >> do something with a file >> except OSError as e: >> if e.errno == ENOENT: >> # file is missing, but that is ok >> # because we treat it like an empty file >> elif ... some other ok situation ... >> else: >> raise >> >> My natural inclination with an ExceptionGroup would be to winnow the >> OSErrors I'm handed, and push the _unhandled_ errors back into the >> original ExceptionGroup. That way, after the various except* clauses, a >> nonempty ExceptionGroup would remain with the unhandled errors, and it >> might perhaps be reraised then. >> >> Cheers, >> Cameron Simpson >> > -- > --Guido (mobile) > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JAVBJENJCBIMC6EVUVZFJJYYPGKRC3R2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Sat, Feb 27, 2021 at 12:35 AM Greg Ewing wrote: > While I don't particularly mind if we get ExceptionGroup, giving > it special magical semantics, and especially new syntax, seems like > bringing a massively overpowered weapon to bear on something that > will only be used rarely. > > Handling multiple exceptions from an ExceptionGroup could be done > using a loop with pattern matching, so is except* really justified? > > It is of course an option to split the PEP into two, add ExceptionGroup first and then ask ourselves if we want except*. We do have some experience with this from the Trio experiments with MultiError though, so we are not starting from scratch. Can you spell out how you see ExceptionGroup handling work with pattern matching? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XZ5EGRGO5LMAO4DYRX2VZLFRBEVHWTIK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Fri, Feb 26, 2021 at 11:43 PM Guido van Rossum wrote: > On Fri, Feb 26, 2021 at 3:18 PM Marco Sulla > wrote: > >> Excuse me if I post here. Maybe is a stupid question: why, instead of >> introducing except*, Python can't extend the functionality of except, >> so it can do what except* would do? >> > > Good question. Here's an example: > ``` > try: > . . . > except OSError as err: > if err.errno != ENOENT: > raise > . . . > ``` > If this would catch ExceptionGroup(OSError), the `err` variable would be > an ExceptionGroup instance, which does not have an `errno` attribute. > > (Irit: Does the PEP answer this question? I couldn't quickly find it in > the rejected ideas. I think it's a reasonable question and we should answer > it, either in the Rationale or in Rejected Ideas.) > Added here: https://github.com/python/peps/pull/1846 > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/K7VCM7OZDIGZDTIZGJZJGWJOBXB65IU4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Sat, Feb 27, 2021 at 12:47 AM Jim J. Jewett wrote:
>
> > Is this not allowed?
>
> >try:
> >try:
> >obj.func()# function that raises ExceptionGroups
> >except AttributeError:
> >logger.info("obj doesn't have a func")
> >except *(AttributeError, SyntaxError):
> >logger.info("func had some problems")
>
> Allowed, but probably in error ... no AttributeError will get through to
> the except * unless it happened inside the except AttributeError handler.
> Did you mean:
>
If obj.func() raises an ExceptionGroup that contains AttributeError then
"except AttributeError" doesn't catch it. So it will get through.
>
> try:
> try:
> obj.func# function that raises ExceptionGroups
> except AttributeError:
> logger.info("obj doesn't have a func")
> obj.func()
> except *(AttributeError, SyntaxError):
> logger.info("func had some problems")
>
> I see this as an argument that the except/except* split is tricky, but I
> don't think it says anything about whether except* clauses should be able
> to see into nested ExceptionGroups ... nor am I at all confident that I
> understood your intent.
>
> -jJ
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XOAB7IJNXOHRL3HRVZ5VZON6MVHOPXB3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SVADA4PBND7ADSGTMC37XCQVNSOQ57LM/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Fri, Feb 26, 2021 at 2:42 PM Nathaniel Smith wrote:
> On Fri, Feb 26, 2021 at 5:05 AM Irit Katriel
> wrote:
> > I'm not sure it's safe to assume that it is necessarily a programming
> error, and that the interpreter can essentially break the program in this
> case.
> > Is this not allowed?
> >
> > try:
> > try:
> > obj.func()# function that raises ExceptionGroups
> > except AttributeError:
> > logger.info("obj doesn't have a func")
> > except *(AttributeError, SyntaxError):
> > logger.info("func had some problems")
>
> I'd be fine with disallowing that. The intuition is that things will
> be simplest if ExceptionGroup is kept as transparent and meaningless
> as possible, i.e. ExceptionGroup(ValueError) and ValueError mean
> exactly the same thing -- "some code inside this block raised
> ValueError" -- and ideally should be processed in exactly the same
> way.
But they don't mean the same thing if this prints hello:
try:
raise ValueError()
except ValueError:
print('hello')
and this raises a RuntimeError:
try:
raise ExceptionGroup("", [ValueError()])
except ValueError:
print('hello')
What am I missing?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZQYMOWWCFBP3TBBNZGD5X45CWJNC4HFF/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
FTR: nobody on this long thread so far has suggested that there are no
valid use cases for `except Exception`.
Thank you for turning to what happens with 'except ValueError' when an
ExceptionGroup[ValueError] is raised, this is important.
I'm not sure it's safe to assume that it is necessarily a programming
error, and that the interpreter can essentially break the program in this
case.
Is this not allowed?
try:
try:
obj.func()# function that raises ExceptionGroups
except AttributeError:
logger.info("obj doesn't have a func")
except *(AttributeError, SyntaxError):
logger.info("func had some problems")
On Fri, Feb 26, 2021 at 5:40 AM Nathaniel Smith wrote:
> On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum wrote:
> >
> > So is "fail-fast if you forget to handle an ExceptionGroup" really a
> feature? (Do we call this out in the PEP?)
> >
> > We may believe that "except Exception" is an abuse, but it is too common
> to dismiss out of hand. I think if some app has e.g. a main loop where they
> repeatedly do something that may fail in many ways (e.g. handle a web
> request), catch all errors and then just log the error and continue from
> the top, it's a better experience if it logs "ExceptionGroup:
> []" than if it crashes.
>
> Yeah, 'except Exception' happens a lot in the wild, and what to do
> about that has been a major sticking point in the ExceptionGroup
> debates all along. I wouldn't say that 'except Exception' is an abuse
> even -- what do you want gunicorn to do if your buggy flask app raises
> some random exception? Crash your entire web server, or log it and
> attempt to keep going? (This is almost your example, but adding in the
> part where gunicorn is reliable and well-respected, and that its whole
> job is to invoke arbitrarily flaky code written by random users.)
> Yury/I/others did discuss the idea of a
> BaseExceptionGroup/ExceptionGroup split a lot, and I think the general
> feeling is that it could potentially work, but feels like a
> complicated and awkward hack, so no-one was super excited about it.
> For a while we also had a compromise design where only
> BaseExceptionGroup was built-in, but we left it non-final specifically
> so asyncio could define an ExceptionsOnlyExceptionGroup.
>
> Another somewhat-related awkward part of the API is how ExceptionGroup
> and plain-old 'except' should interact *in general*. The intuition is
> that if you have 'except ValueError' and you get an
> 'ExceptionGroup(ValueError)', then the user's code has some kind of
> problem and we should probably do something? to let them know? One
> idea I had was that we should raise a RuntimeError if this happens,
> sort of similar to PEP 479. But I could never quite figure out how
> this would help (gunicorn crashing with a RuntimeError isn't obviously
> better than gunicorn crashing with an ExceptionGroup).
>
> == NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS ==
>
> Proposal:
>
> - any time an unwinding ExceptionGroup encounters a traditional
> try/except, then it gets replaced with a RuntimeError whose __cause__
> is set to the original ExceptionGroup and whose first traceback entry
> points to the offending try/except block
>
> - CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right
> *before* we start evaluating 'except' clauses for this try/except
>
> So for example:
>
> If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is
> replaced by a RuntimeError. RuntimeError is an Exception, so the
> 'except Exception' clause catches it. And presumably logs it or
> something. This way your log contains both a notification that you
> might want to switch to except* (from the RuntimeError), *along with*
> the full original exception details (from the __cause__ attribute). If
> it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught
> and that's not so great, but at least you get the RuntimeError to
> point out that something has gone wrong and tell you where?
>
> If an ExceptionGroup(ValueError) hits an 'except ValueError': it
> doesn't get caught, *but* a RuntimeError keeps propagating out to tell
> you you have a problem. And when that RuntimeError eventually hits the
> top of your program or ends up in your webserver logs or whatever,
> then the RuntimeError's traceback will point you to the 'except
> ValueError' that needs to be fixed.
>
> If you write 'except ExceptionGroup': this clause is a no-op that will
> never execute, because it's impossible to still have an ExceptionGroup
> when we start matching 'except' clauses. (We could additionally emit a
> diagnostic if we want.)
>
> If you write bare 'except:', or 'except BaseException': the clause
> always executes (as before), but they get the RuntimeError instead of
> the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can
> retrieve it from the __cause__ attribute. (The only case I can think
> of where this would be useful is if you're writing code that has to
>
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum wrote: > Then we should get some more opinions on this. I think it's the best idea > so far for kindness towards code using `except Exception:`. > I agree that it's the best idea so far. If someone says 'except Exception' they better mean it because we'll believe them, and if someone forgets to handle an ExceptionGroup then they have a bug and that's how it is. > OT: Is ExceptionGroup *really* immutable in the current implementation? As > long as the 'errors' field is a list, I think one could mutate the list > directly. > It's not, but we were going to make it an immutable tuple. > Which brings me to the question, do you have a branch that matches the PEP > yet? > > The implementation matches the PEP less and less every day because the PEP is developing faster than the implementation. But these aren't differences that are more than a technicality to fix (rename things, move an error from runtime to the parser, things like that). The except* implementation is probably pretty close to the PEP because it's the most recent bit. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I473Z5E3FA3SV7MLT74UNPWEG3W2FOCA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Feb 25, 2021 at 9:06 PM Guido van Rossum wrote: > > Hm, a different idea: maybe it's simple enough that we can just add an > example showing how to do this? Then people can tailor that e.g. to use > various traversal orders. (We could just link to the code in traceback.py, > but it probably is full of distractions.) > I've added it here: https://github.com/python/peps/pull/1841 I'd rather do this for now, and add it to the standard library only when we have a better idea about common patterns // best practices for handling ExceptionGroups. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4UDHU7ZMZBGS5V6A62HHYBPG4JCJICQZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
We don't call it out but we talked about this at the sprint. I think the reason we didn't come up with this solution then is that at the time ExceptionGroups in our plans were not immutable, so this would not have been possible (you don't know at construction time what it is going to contain). On Thu, Feb 25, 2021 at 10:08 PM Guido van Rossum wrote: > So is "fail-fast if you forget to handle an ExceptionGroup" really a > feature? (Do we call this out in the PEP?) > > We may believe that "except Exception" is an abuse, but it is too common > to dismiss out of hand. I think if some app has e.g. a main loop where they > repeatedly do something that may fail in many ways (e.g. handle a web > request), catch all errors and then just log the error and continue from > the top, it's a better experience if it logs "ExceptionGroup: > []" than if it crashes. > > There's also code that catches Exception, logs the error and some relevant > data, and then re-raises it. The logged error may go to a more specialized > destination than the generic traceback, and the log message may contain > additional data that's only available in that stack frame. > > So I think there are enough serious use cases that we should do what's > best for those use cases, and not try to lecture users about abuse of the > idiom. > > I don't know what we would have done if we were building Python from > scratch. Possibly we would not have BaseException at all, and the whole > mess would go away. (But there are some good reasons why we introduced > BaseException, so I don't know that that would really be a better overall > experience.) > > --Guido > > > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DJ265GSMTHIIDLKR5JTV3NILOYCLVJDX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Feb 25, 2021 at 5:19 AM Guido van Rossum wrote:
> [Subthread: handling individual errors]
>
> > Rather than iterator, I think we should add a visitor that calls a
> function for each leaf exception,
>
> I agree that we shouldn't add an `__iter__` method. But what if we added a
> separate method that returns an iterator? Then you could write
> ```
> for e in eg.something():
> print("caught", e)
> ```
> This looks cleaner to me than having to write
> ```
> def handler(e):
> print("caught", e)
> eg.something(handler)
> ```
> (Does my bias against callback functions shine through? :-)
>
I hear you. I suggested a visitor to make it a bit awkward to use because
I'm not sure why people should iterate over individual exceptions in an
ExceptionGroup, and I think that by providing an iteration utility we are
implying that this is what you should do.
So let me be more direct instead of proposing passive-aggressive APIs.
Can we take a step back and talk about how we think people would want to
handle ExceptionGroups?
In the rejected ideas section we discuss this a bit, with a reference to
Yury's writeup:
https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284
TL;DR: when you get a group of exceptions from asyncio or the like, you
may want to query it for exception types is contains (with subgroup()), you
may want to format it into a log (with traceback.* methods), but you are
unlikely to care whether there are 1, 2 or 300 ValueErrors. Your program
will probably do the same thing regardless. If you allowed your ValueError
get collected into an ExceptionGroup you already lost the context in which
it happened to it's unlikely that you can make a targeted recovery which is
relevant to this particular exception.
So, what is the use case for iterating over single exceptions?
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/GYSBF6MZDAYG7SMRZGTDXTXY3TYWZAT3/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Thu, Feb 25, 2021 at 5:59 AM Guido van Rossum wrote: > > Here's a potentially alternative plan, which is also complex, but doesn't > require asyncio or other use cases to define special classes. Let's define > two exceptions, BaseExceptionGroup which wraps BaseException instances, and > ExceptionGroup which only wraps Exception instances. (Names to be > bikeshedded.) They could share a constructor (always invoked via > BaseExceptionGroup) which chooses the right class depending on whether > there are any non-Exception instances being wrapped -- this would do the > right thing for split() and subgroup() and re-raising unhandled exceptions. > > Then 'except Exception:' would catch ExceptionGroup but not > BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be > caught (even if there's also e.g. a ValueError among the wrapped errors). > > Pseudo-code: > > class BaseExceptionGroup(BaseException): > def __new__(cls, msg, errors): > if cls is BaseExceptionGroup and all(isinstance(e, Exception) for > e in errors): > cls = ExceptionGroup > return BaseException.__new__(cls, msg, errors) > > class ExceptionGroup(Exception, BaseExceptionGroup): > pass > This could be a valid compromise. We keep the ability to wrap any exception, while we lose the "fail-fast if you forget to handle an ExceptionGroup" feature, which was intended as a kindness towards those who abuse "except Exception". If we adopt this solution then letting an ExceptionGroup escape from code that is not supposed to raise it, is not a fatal error, it's just some exception like any other. So there is no longer a distinction between code that raises ExceptionGroups and code that doesn't. Any code can propagate them, like any code can raise any other exception. Does this mean that more code needs to be aware of the possibility of them showing up? Is that a problem? Maybe this a simpler state of affairs overall. What would we have done here if we were building Python from scratch? Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L3GKLVTZWICQYH5JCJRGQ6K5QX7EWIZX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Jim, On Wed, Feb 24, 2021 at 2:16 PM Jim J. Jewett wrote: > Petr: What about except *(TypeError, ExceptionGroup):? > > Irit: Good question. We should block that as well. > > But https://www.python.org/dev/peps/pep-0654/#backwards-compatibility > seems to explicitly recommend the very similar: > > except (Exception, ExceptionGroup): That's except, not except*. Please include an example for: > > except *ExceptionGroup: pass > There is one here: https://www.python.org/dev/peps/pep-0654/#forbidden-combinations It raises a Runtime Error. The "no subclasses" seems pretty severe, particularly if you can't even use > marker attributes because it isn't clear when a different instance of > container ExceptionGroup will be substituted. > > The justification for this restriction was that .split() had to be be able > to instantiate ... wouldn't it be enough to just say that subclasses need a > compatible __init__ and __new__ for that reason, and then leave it to > consenting adults? What do you mean by compatible? How would one's marker attributes be copied to the new instances? split() currently copies (context, cause, traceback) from the original ExceptionGroup to the new ones. We could have some protocol, but I'm not sure what it would look like. I'm pretty sure there will be a cost, and I'm not sure I understand why no subclassing is a serious enough limitation to justify that. Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NMJSQW5D6O43KMLMOHY6X6Z2PXT5XYFB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Wed, Feb 24, 2021 at 4:39 AM Guido van Rossum wrote: > > OTOH we might reconsider deriving ExceptionGroup from BaseException -- > maybe it's better to inherit from Exception? I don't think the PEP > currently has a clear reason why it must derive from BaseException. I > believe it has to do with the possibility that ExceptionGroup might wrap a > BaseException instance (e.g. KeyboardInterrupt or SystemExit). > That was the reason, and we also said that an ExceptionGroup escaping something that isn't supposed to be raising ExceptionGroups is a bug, so if you call an API that raises ExceptionGroups it is your responsibility to handle them. We could make ExceptionGroup be an Exception, and refuse to wrap anything which is not an Exception. So asyncio either raises KeyboardInterrupt or an ExceptionGroup of user exceptions. Those are quite different directions. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EO6Z3DRWFXCOMEYENLDVZSIN27T24BAI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Wed, Feb 24, 2021 at 4:55 AM Guido van Rossum wrote: However there may be an omission in the PEP -- what if we want to do > something special for each suberror? If we just iterate over `eg.errors` we > would have to do something recursive for exceptions wrapped inside multiple > nested groups. We could add a helper method to ExceptionGroup that iterates > over suberrors, flattening nested groups. But we'd need to do something > special to recover the tracebacks there (groups share the common part of > the traceback). Or we could say "if you need the traceback you're going to > have to write something recursive" -- like we have to do in traceback.py. > But we might as well generalize whatever we're doing there. (Irit: I > suspect there's a piece of API design we missed here.) > This is mentioned briefly where we talk about why ExceptionGroup is not iterable: https://www.python.org/dev/peps/pep-0654/#the-exceptiongroup-api We left it out pending a use case. Rather than iterator, I think we should add a visitor that calls a function for each leaf exception, plus a utility that returns the traceback of a leaf exception (as a single list, copying frames from the tracebacks of ExceptionGroups so that it's not destructive.) This way the expensive traceback construction is happening explicitly when it is needed. I think accessing one exception at a time to do something other than formatting it is more likely to be overused than actually needed. We did that in an earlier version of the PEP for the "filter OSErrors by type" example, which we did with iteration and now do with subgroup: try: low_level_os_operation() except *OSerror as errors: raise errors.subgroup(lambda e: e.errno != errno.EPIPE) from None ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5VX7AG266VCSZUV4QYXSWD77W7LOC75X/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Petr, Thank you for your careful reading and encouragement. > The `ExceptionGroup` class is final, i.e., it cannot be subclassed. > > What's the rationale for this? > ExceptionGroup.subgroup()/split() need to create new instances, and subclassing would make that complicated if we want the split results to have the same type. > It is possible to catch the ExceptionGroup type with except, but not > with except* because the latter is ambiguous > > What about `except *(TypeError, ExceptionGroup):`? > Good question. We should block that as well. > > Motivation: Errors in wrapper code > > This use case sticks out a bit: it's the only one where ExceptionGroup > doesn't represent joining equivalent tasks. > Consider code similar to bpo-40857: > >try: >with TemporaryDirectory() as tempdir: >os.rmdir(tempdir) >n = 1 / 0 >except ArithmeticError: ># that error can be safely ignored! >pass > > Instead of a FileNotFoundError with ArithmeticError for context you'd > now get an ExceptionGroup. Neither is handled by `except > ArithmeticError`. Where is the win? > I agree, if TemporaryDirectory() were ever to adopt ExceptionGroups then it will probably be through a new API (like an opt-in parameter) to avoid breaking current code. Users of such a TemporaryDirectory would need to wrap the calls with try-except* (see the example in my previous reply to Steve). > > Motivation: Multiple failures when retrying an operation > > This is somewhat similar to the TemporaryDirectory, except there's no > `with` block that feels like it should be "transparent" w.r.t. user errors. > If I currently have: > > try: > create_connection(*addresses) > except (Timeout, NetworkNotConnected): > # that's OK, let's try later > pass > > what should happen after Python 3.10? Apart from adding a new function, > I can see two possible changes: > - create_connection() starts always raising ExceptionGroup on error, > breaking backwards compatibility in the error case > - create_connection() starts only raising ExceptionGroup only for 2+ > errors, breaking backwards compatibility in the 2+ errors case > Both look like heisenbug magnets. IMO, the second one is worse; "code > which is now *potentially* raising ExceptionGroup" (as mentioned in the > Backwards Compatibility section; emphasis mine) should be discouraged. > > Arguably, this here is a problem with the create_connection function: > the PEP adds a better way how it could have been designed, and that is > virtuous. Still, having it in Motivation might be misleading. > I agree. Raising ExceptionGroups is an API breaking change, for any library. No function should just start raising ExceptionGroups. > long term plan to replace `except` by `catch` > > Whoa! Is that a real plan? > In the rejected ideas section, anything goes! Irit > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R7ONTCH5MUWKBOPHY2QYKVNPOT5MXPYZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Caleb,
On Tue, Feb 23, 2021 at 11:05 PM Caleb Donovick
wrote:
> What is the motivation for returning `None` on empty splits? I feel like
> this creates an unnecessary asymmetry. I don't personally have a use
> case for the this feature so I may be missing something but it seems like
> it would force an annoying pattern:
>
Split is used by the interpreter to implement except*. It needs to check
whether the first value of the split is empty or not (this determines
whether the except* clause executes).
This is more efficiently done with None. And then there is creating an
object that you don't need.
The intention is that you use except* rather than do this:
> ```
> try:
> foo()
> except ExceptionGroup as eg:
> g1, g2 = eg.split(predicate)
> for e in g1:
> handle_true(e)
> for e in g2:
> handle_false(e)
> ```
>
(as an aside - exception groups ended up not being iterable - that is one
of the rejected ideas, see explanation there).
> Also this creates an subtle difference with subgroup:
>
> ```
> g1, g2 = eg.split(predicate)
> h1, h2 = eg.subgroup(predicate), eg.subgroup(lambda e: not predicate(e))
> assert g1 == h1 and g2 == h2 # only true if `None not in {g1, g2}`
> ```
>
An empty subgroup should also return None. I see this is not mentioned in
the PEP, so I will add it. Thanks.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PW25VAAF3J3WQFNBC6NBYJXQIK63NTQO/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Steve,
Thank you for trying out the implementation. Please do let me know about
bugs you find.
This part of your code looks good:
--- a/Lib/tempfile.py
> +++ b/Lib/tempfile.py
> @@ -819,8 +819,14 @@ def __repr__(self):
> def __enter__(self):
> return self.name
>
> -def __exit__(self, exc, value, tb):
> -self.cleanup()
> +def __exit__(self, exc_cls, exc_value, tb):
> +try:
> +self.cleanup()
> +except Exception as clean_exc:
> +if exc_value is not None:
> +raise ExceptionGroup('Exception occurred during cleanup',
> [exc_value, clean_exc])
> +else:
> +raise
>
> def cleanup(self):
> if self._finalizer.detach():
>
def do_some_stuff():
with tempfile.TemporaryDirectory() as td:
os.rmdir(td)
pathlib.Path(td).write_text("Surprise!")
1/0
Then, use it like this:
if __name__ == '__main__':
try:
do_some_stuff()
except *NotADirectoryError as e:
print("Fail Site 2", repr(e))
except *Exception as e:
print("Fail Site 3", repr(e))
else:
print("No error")
Output:
Fail Site 2 ExceptionGroup('Exception occurred during cleanup',
[NotADirectoryError(20, 'The directory name is invalid')])
Fail Site 3 ExceptionGroup('Exception occurred during cleanup',
[ZeroDivisionError('division by zero')])
This works whether do_some_stuff raises naked exceptions or
ExceptionGroups, because "except *T" catches a naked T (and wraps it in an
ExceptionGroup).
So if I comment out the 1/0 I get:
Fail Site 2 ExceptionGroup('', (NotADirectoryError(20, 'The directory
name is invalid'),))
There is no need for it to always raise ExceptionGroup. It can propagate
the user's exception as is if it has no exceptions to add. But the caller
needs to assume that it may raise an ExceptionGroup, and use except*.
Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/AI7YFJK7KSCVNBAMAKQWMLSIGTT3OJ5X/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi Damian, While I agree that there are a handful of use cases for catching all Exceptions, if you look at some of the 5 million hits you found in github, the vast majority are not such cases. They are mostly try/except blocks that wrap a particular operation (a dict access, a raw_input call, etc). You don't want to automatically translate "except Exception" to something that involves ExceptionGroups, because most of the time it is not needed -- the code in the body can't raise ExceptionGroups. Your first email on this thread showed that you were able to quite quickly, just from what's already in the PEP, think of how to do what you need for your valid "except Exception" use case. My feeling is that people who need it will be able to do it quite easily. I'm more worried about people assuming that they need it when they don't. Irit On Tue, Feb 23, 2021 at 8:00 PM Damian Shaw wrote: > Hi Irit, > > Catching exceptions like this is an extremely common pattern in the real > world, e.g. this pattern has over 5 million GitHub matches: > https://github.com/search?l==%22except+Exception%22+language%3APython=code > > How common it is aside there are also many valid use cases for this > pattern, e.g. logging a final unhandled exception of a script, catching > exceptions inside an orchestration framework, exploring code where the list > of exceptions is unknown, etc. > > A description from the PEP on how to handle this kind of pattern, > especially for code that must support multiple versions of Python, would be > extremely helpful. > > Damian > >> >> ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5SOONPYY3COVU43QMOG37EBMHSE4JFSW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission
On Tue, Feb 23, 2021 at 3:49 PM Damian Shaw wrote: > > Firstly, if I have a library which supports multiple versions of Python > and I need to catch all standard exceptions, what is considered the best > practise after this PEP is introduced? > > Currently I might have code like this right now: > try: > ... # Code > except Exception as e: > ... # Logic to handle exception > > Hi Damian, Catching all exceptions in this way is not a common pattern. Typically you have an idea what kind of exceptions you expect to get from an operation, and which of those exceptions you are interested in handling. Most of your try/except blocks will be targeted, like handling KeyError from d[k], an operation that will never raise an ExceptionGroup. ExceptionGroups will only be raised by specific APIs which will advertise themselves as such. We don't expect that there will be a need for blanket migrations of "except T" to "except (T, ExceptionGroup)" to "except *T". Irit ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TWEGKU7H5GC3I4WHNVBQ2EPWQMSP633Q/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 654 -- Exception Groups and except* : request for feedback for SC submission
Hi all, We would like to request feedback on PEP 654 -- Exception Groups and except*. https://www.python.org/dev/peps/pep-0654/ It proposes language extensions that allow programs to raise and handle multiple unrelated exceptions simultaneously, motivated by the needs of asyncio and other concurrency libraries, but with other use cases as well. * A new standard exception type, ExceptionGroup, to represent multiple exceptions with shared traceback. * Updates to the traceback printing code to display (possibly nested) ExceptionGroups. * A new syntax except* for handling ExceptionGroups. A reference implementation (unreviewed) can be found at: https://github.com/iritkatriel/cpython/pull/10 Thank you for your help Kind regards Irit, Yury & Guido ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L5Q27DVKOKZCDNCAWRIQVOZ5DZCZHLRM/ Code of Conduct: http://python.org/psf/codeofconduct/
