For what it's worth, I've increasingly considered creating a context in a generator (sync or async) to be an antipattern. To this end, in my case, I moved creation of the context outside of the generator. This resulted in a more explicit contract, which actually simplified reasoning about its behavior.
On Sun, 2021-02-07 at 16:18 -0800, Richard Levasseur wrote: > I, too, think I ran into basically the same situation as Paul: An > outer async generator calls an inner async generator. The outer > doesn't fully consume the inner. The inner uses an async context > manager. The inner CM isn't exited when expected and resources aren't > released when required/expected. > > FWIW, the analysis of PEP 533 seems spot on -- for synchronous code, > while still possible, it's harder to trigger thanks to the GC tricks. > For asynchronous code, it easily (and subtly) results in resources > not being released when expected. Or perhaps never at all? IIUC, a > long-lived server might just accumulate unfinished async generators > indefinitely, or at least until some OS resource was exhausted. > > In my particular case, I have thousands of network requests to make, > each of which is potentially large and/or slow to complete in its > entirety, so I'm taking extra care to break early to minimize > resource usage to both client and servers. I spent about an entire > day trying to figure why the synchronous version of my code Just > Worked, while the asynchronous version simply refused to close things > when I was done with them. It was only after I started looking at > third party libraries and stumbled upon a reference to PEP 533 and a > detailed blog post from 5 years ago that I had a clue what was going > on. My point being, the feature PEP 533 describes would have made > things Just Work in line with my expectations. > > > On Tue, Jan 5, 2021 at 8:35 AM Stestagg <stest...@gmail.com> wrote: > > For what it's worth, this recent issue: > > https://bugs.python.org/issue42762 is somewhat related to the non- > > async aspect of 533. (generator func getting closed during __del__ > > and causing odd side-effects at seemingly random points in the > > execution flow) > > > > The example code provided in that issue is contrived, and involves > > catching unconditional exceptions and recursing, but following the > > code flow in that situation feels unreasonably hard, and deciding > > if it's a python bug or not is even harder. > > > > I'm not sure how easy an implementation of 533 would be that > > doesn't break a lot of existing code, but if one could be made, it > > might help make generator lifecycles more predictable. > > > > > > > > > > On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran > > <sent...@uthcode.com> wrote: > > > Hi Paul, > > > > > > > Per PEP 525, I can call aclose coroutine method to cleanup the > > > generator, but > > > > it requires the code iterating to be aware that that closing > > > the generator is > > > > necessary. > > > > > > How about treating this as a bug for the specific use case that > > > you mentioned, > > > rather than a complete addition of " PEP 533 -- Deterministic > > > cleanup for > > > iterators". I am not certain why PEP 533 was deffered. > > > > > > I'd also suggest a documentation update if the details of aclose > > > requirement is > > > mentioned only in a PEP and not in the documentation. > > > > > > Thank you, > > > Senthil > > > _______________________________________________ > > > Python-Dev mailing list -- python-dev@python.org > > > To unsubscribe send an email to python-dev-le...@python.org > > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > > Message archived at > > > > > > https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7DH5G6YR3AQHZTQKHBQF46YXLC/ > > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-le...@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > > https://mail.python.org/archives/list/python-dev@python.org/message/VB64QJAYU2CCLOHRCBUIQOHMS2DIQQCC/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/D7CO5ZMIPT3R4YO3OCWGAYG5PFC5S53I/ > Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4FCYB63LCWI5D7AGRRDHEEXE5IIB2SAZ/ Code of Conduct: http://python.org/psf/codeofconduct/
