[issue41756] Do not always use exceptions to return result from coroutine

Petr Viktorin added the comment: that is, bpo-44727 -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41756] Do not always use exceptions to return result from coroutine

Petr Viktorin added the comment: Hello! This change added an enum to the stable ABI: typedef enum { PYGEN_RETURN = 0, PYGEN_ERROR = -1, PYGEN_NEXT = 1, } PySendResult; Adding new values to enums might break the stable ABI in some (admittedly rare) cases; so

[issue41756] Do not always use exceptions to return result from coroutine

Change by Andrew Svetlov : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list

[issue41756] Do not always use exceptions to return result from coroutine

Andrew Svetlov added the comment: Can we close the issue? -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: New changeset cfb0f57ff876ab3d04ff144f19eda58844981643 by Vladimir Matveev in branch 'master': bpo-41756: Export PyGen_Send and wrap it in if-defs (#22677) https://github.com/python/cpython/commit/cfb0f57ff876ab3d04ff144f19eda58844981643 --

[issue41756] Do not always use exceptions to return result from coroutine

Change by Vladimir Matveev : -- pull_requests: +21649 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/22677 ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > Also, in Include/abstract.h it should only be available for limited C API >= > 3.10: Vladimir, could you please submit a PR? > We now should discuss the behavior of PySend_Iter() if value is NULL. It is > not documented, the current behavior differs from

[issue41756] Do not always use exceptions to return result from coroutine

Change by Yury Selivanov : -- status: closed -> open ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: Few things I forget about. The new C API function should be exported in PC/python3dll.c. Also, in Include/abstract.h it should only be available for limited C API >= 3.10: #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A ... #endif We now

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: With the latest PR now merged this issue can be closed. Please reopen if there are any other action items left. -- stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Change by Vladimir Matveev : -- pull_requests: +21639 pull_request: https://github.com/python/cpython/pull/22663 ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Mark Shannon added the comment: What's the difference between removing it and making it properly private (i.e. static)? It's an irrelevant detail whether the code is inlined or in a helper function. -- ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: > Don't forget to remove PyGen_Send() That's the function that allows sending data into a generator. It's also used internally by "PyIter_Send()". Are you really suggesting to remove it, or to make it underscore-private again? (As it was before.) --

[issue41756] Do not always use exceptions to return result from coroutine

Mark Shannon added the comment: Vladimir, Thanks for adding PyIter_Send(). Don't forget to remove PyGen_Send() :) -- ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: New changeset 037245c5ac46c3436f617a1f5d965929754be239 by Vladimir Matveev in branch 'master': bpo-41756: Add PyIter_Send function (#22443) https://github.com/python/cpython/commit/037245c5ac46c3436f617a1f5d965929754be239 --

[issue41756] Do not always use exceptions to return result from coroutine

Change by Vladimir Matveev : -- pull_requests: +21473 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/22443 ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: No, I don't think so but I can definitely make one. A few questions first: - having PySendResult as a result type of PyIterSend seems ok, however prefix for each concrete value (PYGEN_*) is not aligned with the prefix of the function itself (PyIter_) -

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: No, I meant a function which combines PyGen_Send, tp_iternext and _PyGen_FetchStopIterationValue. Was not it in your PR? -- ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: Serhiy, AFAIR PyIter_Send in my PR appear only as a rename from placeholder `Name_TBD` and it still was specific to PyGenObjects. Do you mean something that was listed in https://bugs.python.org/msg377007 ? --

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: I agree that we should add PyIter_Send() (the name is discussable, but PyIter_Send LGTM) which should support iterators and objects with the send() method. It would be much more useful for user, and can replace PyGen_Send() in the interpreter core code.

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: New changeset 6c33385e3a1dce31d7b9037eebfc584075795dba by Serhiy Storchaka in branch 'master': bpo-41756: Refactor gen_send_ex(). (GH-22330) https://github.com/python/cpython/commit/6c33385e3a1dce31d7b9037eebfc584075795dba --

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > If you disagree with me, please say why, don't just merge the PR. Apologies, Mark. I didn't intend to merge something bypassing your opinion; just missed your comment between reviewing multiple PRs in a few unrelated repos. I'm sorry. On the actual

[issue41756] Do not always use exceptions to return result from coroutine

Mark Shannon added the comment: Yury, Why was the PR merged with a new API function `PyGen_Send`? I explicitly said that any new API function should *not* start with `PyGen`, nor should any function rely on generators and async "coroutines" sharing the same memory layout. If you disagree

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: Vladimir, could you please submit a PR to update 3.10/whatsnew? Need to mention both the new C API and the new perf boost in relevant sections. -- ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: PR 22330 refactors gen_send_ex(), making it similar to the new PyGen_Send(). -- status: closed -> open ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Change by Serhiy Storchaka : -- pull_requests: +21374 pull_request: https://github.com/python/cpython/pull/22330 ___ Python tracker ___

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: Thanks, Vladimir! Also huge thanks to Mark, Serhiy, and Stefan. If there are any nits to fix let's do that in follow up PRs. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: New changeset 2b05361bf7cbbd76035206fd9befe87f37489f1e by Vladimir Matveev in branch 'master': bpo-41756: Introduce PyGen_Send C API (GH-22196) https://github.com/python/cpython/commit/2b05361bf7cbbd76035206fd9befe87f37489f1e --

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: Yes, it should be -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > Sounds like a good middleground to start: add ``PySendResult `` and > `PySendResult PyGen_Send(PyGenObject*, PyObject* PyObject**)` specific to > generators and coroutines. Yes, it seems that everybody agreed on that. I can give the PR another review --

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: Sounds like a good middleground to start: add ``PySendResult `` and `PySendResult PyGen_Send(PyGenObject*, PyObject* PyObject**)` specific to generators and coroutines. Subsequent changes could introduce `PySendResult PyIter_Send(PyObject*, PyObject*,

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > I would also have preferred a more type specific function, but yeah, as long > as the types for which the function would normally be used are special cased > early enough in the implementation, it makes no big difference. Maybe add two API funcs:

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: We introduced _PyObject_LookupAttr() and _PyObject_GetMethod() for similar purposes. And they have similar signatures. Although they all are private. -- ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: BTW, just to give this a house number, I remember having measured a performance improvement of up to 70% at some point when switching from "generators always raise a StopIteration at the end" to "generators just return NULL" in Cython. For short-running

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: I would also have preferred a more type specific function, but yeah, as long as the types for which the function would normally be used are special cased early enough in the implementation, it makes no big difference. Fine with me, too. --

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: I would be fine even with a generator-specific API as a first step, for simplicity. But the end goal is to support all generator-like objects. It is much more useful for end users and does not have drawbacks. Enum for result seems not necessary (other

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > Does it sound correct? It does, but given the amount of back and forth on this, I'd wait for Serhiy and Stefan to confirm if they're OK. IMO the `PyIter_Send` name is OK (given how generic the implementation will be) and returning an enum, while a tad

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: so to summarize: Proposed function signature: ``` PySendResult PyIter_Send(PyObject *obj, PyObject *arg, PyObject **result); ``` For generators/coroutines function will delegate to specialized implementation that does not raise StopIteration exception For

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > I guess `PyIter_Send` would imply that this function should work for all > inputs (like in https://bugs.python.org/msg377007) which also sounds > reasonable. I like `PyIter_Send` (and why I initially proposed it myself) because it will also work with the

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: I guess `PyIter_Send` would imply that this function should work for all inputs (like in https://bugs.python.org/msg377007) which also sounds reasonable. -- ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

Mark Shannon added the comment: I agree with Serhiy, that `PyGen_` is a bad prefix. Unless a function takes generator objects and *only* generators objects, then it shouldn't have a `PyGen` prefix. The API function is the C equivalent of obj.send(val). The first parameter is an object.

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: There are other abstract object APIs: PyNumber, PySequence, PyMapping, etc. In particularly PyIter_Next() works with the iterator protocol, there is no single iterator class. Seems PyGen_* API is related to concrete class, but we can introduce new

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > I think it should be specific to generators and coroutines. Calling > `PyObject_CallMethodIdOneArg(coro, _send, arg);` and interpreting > exceptions to emulate the low level API seems a bit too much. To add to my point: typically higher-level APIs go

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > Since coroutines inherit the generator protocol more or less, I think > "PyGen_Send()" is a more suitable name, better than "PyCoro_Send()". OK, +1. PyGen_Send it is. > Also should it be specific to generators/coroutines and accept PyGenObject* I think

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: Also should it be specific to generators/coroutines and accept PyGenObject* or should it try to handle multiple cases and expose the result for them in uniform way, i.e. ``` if (PyGen_CheckExact(gen) || PyCoro_CheckExact(gen)) { // use

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: I'm happy to see this moving forward. Not convinved of the "PyIter_Send()" name, though. I don't consider this part of the iterator protocol. It's specific to generators and coroutines. Cython would probably guard its usage by "PyGen_CheckExact()" or

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: > As for returned value, I propose to return -1 in case of error, 1 for yielded > value and 0 for returned value (i.e. define PYGEN_RETURN = 0, PYGEN_YIELD = 1 > and PYGEN_ERROR = -1, but without exposing public names). Sure, that works. --

[issue41756] Do not always use exceptions to return result from coroutine

Serhiy Storchaka added the comment: As for returned value, I propose to return -1 in case of error, 1 for yielded value and 0 for returned value (i.e. define PYGEN_RETURN = 0, PYGEN_YIELD = 1 and PYGEN_ERROR = -1, but without exposing public names). It would be uniform with other C API:

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: Mark, Stefan, I don't want this to be stale so I propose to move with my suggestions: 1. We make the new API public. Mark, if you have objections to that - please elaborate with some details. IMO, the corresponding Python API is long public and there's no

[issue41756] Do not always use exceptions to return result from coroutine

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: @Mark > Mark Shannon wrote: I don't think [the C-API function] should be public, as a > possible further improvement is to stop passing exceptions through a side > channel, but in result. Maybe we don't want to do that, but lets' not add to > the (already

[issue41756] Do not always use exceptions to return result from coroutine

Vladimir Matveev added the comment: If I understand proposed shape of API correctly - it was not supposed to return exception via "result" so contract for new `PyGen_Send` function is something like: Return value | result | Comment -

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: Copying some of the design discussion from the PR here (https://github.com/python/cpython/pull/22196/files#r486730457), because it belongs into the ticket. Yury Selivanov proposed to add a new C-API function for this (naming changes by me): typedef

[issue41756] Do not always use exceptions to return result from coroutine

Change by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue41756] Do not always use exceptions to return result from coroutine

Stefan Behnel added the comment: Big +1 from me, too, for the same reasons Yury gave. -- nosy: +scoder ___ Python tracker ___ ___

[issue41756] Do not always use exceptions to return result from coroutine

Yury Selivanov added the comment: Big +1 from me. This is something I always wanted to do myself (since the time of PEP 492 & 525 implementations) and I think this is a necessary change. It's great that this isn't just a C API UX improvement but also yields a big perf improvement.

[issue41756] Do not always use exceptions to return result from coroutine

Change by Vladimir Matveev : -- keywords: +patch pull_requests: +21255 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22196 ___ Python tracker

[issue41756] Do not always use exceptions to return result from coroutine

New submission from Vladimir Matveev : Currently async functions are more expensive to use comparing to their sync counterparts. A simple microbenchmark shows that difference could be quite significant: ``` import time def f(a): if a == 0: return 0 return f(a - 1) async def