New submission from Joshua Oreman :
The current async generator finalization hooks are per-thread, but sometimes
you want different async generator semantics in different async tasks in the
same thread. This is currently challenging to implement using the thread-level
hooks. I'm proposing a small backwards-compatible change to the existing async
generator hook semantics in order to better support this use case. I'm seeking
feedback on the proposal and also on how "major" it would be considered. Does
it need a PEP? If not, does it need to wait for 3.10 or could it maybe still
make 3.9 at this point?
TL;DR: if the firstiter hook returns a callable, use that as the finalizer hook
for this async generator instead of using the thread-level finalizer hook.
== Why would you want this? ==
The use case that brought me here is trio-asyncio, a library that allows
asyncio and Trio tasks to coexist in the same thread. Trio is working on adding
async generator finalization support at the moment, which presents problems for
trio-asyncio: it wouldn't work to finalize asyncio-flavored async generators as
if they were Trio-flavored, or vice versa. It's easy to tell an async
generator's flavor from the firstiter hook (just look at which flavor of task
is running right now), but hard to ensure that the corresponding correct
finalizer hook is called (more on this below).
There are other possible uses as well. For example, one could imagine writing
an async context manager that ensures all async generators firstiter'd within
the context are aclose'd before exiting the context. This would be less verbose
than guarding each individual use of an async generator, but still provide more
deterministic cleanup behavior than leaving it up to GC.
== Why is this challenging to implement currently? ==
Both of the above use cases want to provide a certain async generator
firstiter/finalizer behavior, but only within a particular task or tasks. A
task-local firstiter hook is easy: just install a thread-local hook that checks
if you're in a task of interest, calls your custom logic if so or calls the
previous hook if not. But a task-local finalizer hook is challenging, because
finalization doesn't necessarily occur in the same context where the generator
was being used. The firstiter hook would need to remember which finalizer hook
to use for this async generator, but where could it store that information?
Using the async generator iterator object as a key in a regular dictionary will
prevent it from being finalized, and as a key in a WeakKeyDictionary will
remove the information before the finalizer hook can look it up (because
weakrefs are broken before finalizers are called). About the only solution I've
found is to store it in the generator's f_locals dict, but that's not very ap
pealing.
== What's the proposed change? ==
My proposal is to allow the firstiter hook to return the finalizer hook that
this async generator should use. If it does so, then when this async generator
is finalized, it will call the returned finalizer hook instead of the
thread-level one. If the firstiter hook returns None, then this async generator
will use whatever the thread-level finalizer was just before firstiter was
called, same as the current behavior.
== How disruptive would this be? ==
Async generator objects already have an ag_finalizer field, so this would not
change the object size. It's just providing a more flexible way to determine
the value of ag_finalizer, which is currently not accessible from Python.
There is a theoretical backwards compatibility concern if any existing
firstiter hook returns a non-None value. There wouldn't be any reason to do so,
though, and the number of different users of set_asyncgen_hooks() currently is
likely extremely small. I searched all of Github and found only asyncio, curio,
uvloop, async_generator, and my work-in-progress PR for Trio. All of these
install either no firstiter hook or a firstiter hook that returns None.
The implementation would likely only be a handful of lines change to
genobject.c.
--
components: asyncio
messages: 371053
nosy: Joshua Oreman, asvetlov, njs, yselivanov
priority: normal
severity: normal
status: open
title: Proposed tweak to allow for per-task async generator semantics
type: enhancement
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue40916>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
