[issue46843] PersistentTaskGroup API

Guido van Rossum added the comment: Okay. -- stage: -> resolved status: open -> closed ___ Python tracker ___ ___

[issue46843] PersistentTaskGroup API

Andrew Svetlov added the comment: I think we should close the PR now. I'm open to the discussion resurrection in Python 3.12 or 3.13, when aiotools implementation will be battle-tested. -- ___ Python tracker

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: I have released the new version of aiotools with rewritten TaskGroup and PersistentTaskGroup. https://aiotools.readthedocs.io/en/latest/aiotools.taskgroup.html aiotools.TaskGroup has small additions to asyncio.TaskGroup: a naming API and `current_taskgroup`

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: I have updated the PersistentTaskGroup implementation referring asyncio.TaskGroup and added more detailed test cases, which works with the latest Python 3.11 GitHub checkout. https://github.com/achimnol/aiotools/pull/36/files Please have a look at the class

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Short summary: PersistentTaskGroup shares the followings from TaskGroup: - It uses WeakSet to keep track of child tasks. - After exiting the async context manager scope (or the shutdown procedure), it ensures that all tasks are complete or cancelled.

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: > And just a question: I'm just curious about what happens if belonging tasks > see the cancellation raised from their inner tasks. Sibling tasks should not > be cancelled, and the outer task group should not be cancelled, unless the > task group itself has

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: > As for errors in siblings aborting the TaskGroup, could you apply a wrapper > to the scheduled coroutines to swallow and log any errors yourself? Yes, this could be a simplest way to implement PersistentTaskGroup if TaskGroup supports "persistent" option to

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Good to hear that TaskGroup already uses WeakSet. When all tasks finish, PersistentTaskGroup should not finish and wait for future tasks, unless explicitly cancelled or shutdown. Could this be also configured with asyncio.TaskGroup? I'm also ok with adding a

[issue46843] PersistentTaskGroup API

Tin Tvrtković added the comment: The asyncio TaskGroup already uses a WeakSet for its children, so it's suitable for long-lived use. As for errors in siblings aborting the TaskGroup, could you apply a wrapper to the scheduled coroutines to swallow and log any errors yourself? Apart from the

[issue46843] PersistentTaskGroup API

Guido van Rossum added the comment: (FWIW I would close this issue but I'll wait to see if @asvetlov has something to add.) -- ___ Python tracker ___

[issue46843] PersistentTaskGroup API

Guido van Rossum added the comment: The implementation of asyncio.TaskGroup isn't all that complicated (and the new "cancel count" API helps). I recommend that you build one that satisfies your requirements yourself, or convince the authors of some other package like Quattro or aiotools to

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Anoter case: https://github.com/lablup/backend.ai-manager/pull/533 https://github.com/lablup/backend.ai-agent/pull/341 When shutting down the application, I'd like to explicitly cancel the shielded tasks, while keep them shielded before shutdown. So I inserted

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Here is one another story. When handling message queues in distributed applications, I use the following pattern frequently for graceful shutdown: * Use a sentinel object to signal the end of queue. * Enqueue the sentinel object when: - The server is shutting

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: I ended up with the following conclusion: - The new abstraction should not cancel sibling tasks and itself upon unhandled execption but loudly report such errors (and the fallback error handler should be customizable). - Nesting task groups will give additional

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: This particular experience, https://github.com/lablup/backend.ai-agent/pull/331, has actually motivated me to suggest PersistentTaskGroup. The program subscribes the event stream of Docker daemon using aiohttp as an asyncio task, and this should be kept

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: @gvanrossum As you mentioned, the event loop currently plays the role of the top-level task group already, even without introducing yet another top-level task. For instance, asyncio.run() includes necessary shutdown procedures to cancel all belonging

[issue46843] PersistentTaskGroup API

Guido van Rossum added the comment: Could you just have a global task group that owns these long-running tasks? It could be embedded in a "toplevel" task that is created using asyncio.create_task() (which won't be deprecated). To shut down all long-running tasks at the end, just cancel that

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: @yselivanov @asvetlov I think this API suggestion would require more refining and discussion in depths, and probably it may be better to undergo the PEP writing and review process. Or I might need to have a separate discussion thread somewhere else (maybe

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Some search results from cs.github.com with the input "asyncio task weakset", which may be replaced/simplified with PersistentTaskGroup: - https://github.com/Textualize/textual/blob/38efc821737e3158a8c4c7ef8ecfa953dc7c0ba8/src/textual/message_pump.py#L43 -

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: Example use cases: * Implement an event iteration loop to fetch events and dispatch the handlers depending on the event type (e.g., WebSocket connections, message queues, etc.) - https://github.com/aio-libs/aiohttp/pull/2885 -

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: I think people may ask "why in stdlib?". My reasons are: - We are adding new asyncio APIs in 3.11 such as TaskGroup, so I think it is a good time to add another one, as long as it does not break existing stuffs. - I believe that long-running task sets are

[issue46843] PersistentTaskGroup API

Joongi Kim added the comment: So I have more things in mind. Basically PersistentTaskGroup resemble TaskGroup in that: - It has the same "create_task()" method. - It has an explicit "cancel()" or "shutdown()" method. - Exiting of the context manager means that all tasks of it have either

[issue46843] PersistentTaskGroup API

New submission from Joongi Kim : I'm now tracking the recent addition and discussion of TaskGroup and cancellation scopes. It's interesting! :) I would like to suggest to have a different mode of operation in asyncio.TaskGroup, which I named "PersistentTaskGroup". AFAIK, TaskGroup targets