[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

I have added more about my stories in bpo-46843.

I think the suggestion of implicit taskgroup binding with the current 
asyncio.TaskGroup has no point but it would have more meaning with 
PersistentTaskGroup.

So, if we treat PersistentTaskGroup as a "nested, hierarchical virtual event 
loop" to repeat and group shutdown procedures for different task sets 
separately, the point may look a little bit clearer.

It is more like assigning a virtual event loop to different modules and 
libraries, while keeping the behavior of asyncio.create_task() same.  The 
difference is that the caller controls when these virtual loops are terminated 
and in what order.

Does this make sense better?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Guido van Rossum]

Guido van Rossum  added the comment:

-1. Libraries that manage their own tasks should continue to do so, until they 
are ready to adopt TaskGroup. Trying to "own" (and wait for) tasks created by a 
library sounds like a recipe for disaster if the library wasn't expecting that.

Do you have a specific use case or scenario that has bitten you in the past? If 
you want to continue to argue for this feature we would need specifics (not a 
link to reams of code but a clear story telling of a problem you've encountered 
in real life in the past that your proposal might solve).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

Ok, let me be clear: Patching asyncio.create_task() to support this opt-in 
contextual task group binding is not an ultimate goal of this issue.  If it 
becomes possible to override/extend the task factory at runtime with any event 
loop implementation, then it's ok to implement this feature request as a 
3rd-party library.  I also don't want to bloat the stdlib with version-specific 
branches, if there are alternative ways to achieve the same goal.  I just 
wanted to check out your opinons and potential alternative approaches to 
implement it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I personally don't think that the described opt-in trick should be a part of 
asyncio.
A third-party library that patches asyncio.create_task() can be a useful thing 
though during the transition period.
We even cannot deprecate asyncio.create_task() right now: the minimal supported 
Python 3.7 thas now alternative, writing cross-version code is overcomplicated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

Ah, and this use case also requires that TaskGroup should have an option like 
`return_exceptions=True` which makes it not to cancel sibling tasks upon 
unhandled exceptions, as I suggested in PersistentTaskGroup (bpo-46843).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

An example would be like:

tg = asyncio.TaskGroup()
...
async with tg:
  with asyncio.TaskGroupBinder(tg):  # just a hypothetical API
asyncio.create_task(...) # equivalent to tg.create_task(...)
await some_library.some_work()   # all tasks are bound to tg
  asyncio.create_task(...)   # fire-and-forget (not bound to tg)

If TaskGroup supports enumeration/counting of its own tasks and asyncio allows 
enumeration of TaskGroups just like asyncio.Task.all_tasks(), we could extend 
aiomonitor to provide per-taskgroup statistics.

In my projects, we have multiple cases to find and fix bugs in customer sites 
using aiomonitor and I'm willing to improve aiomonitor to support task groups 
as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

My propsal is to opt-in the taskgroup binding for asyncio.create_task() under a 
specific context, not changing the defautl behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Andrew Svetlov]

Andrew Svetlov  added the comment:

-1

Now bare `create_task()` does fire-and-forget action.
After the proposed change it will fail loudly.
Even if this behavior is better it is not backward compatible.
People start blaming and asking "how to return everything back?"

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

It is also useful to write debugging/monitoring codes for asyncio applications. 
 For instance, we could "group" tasks from different libraries and count them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

Conceptually it is similar to replace malloc using LD_PRELOAD or 
LD_LIBRARY_PATH manipulation.  When I cannot modify the executable/library 
binaries, this allows replacing the functionality of specific functions.

If we could assign a specific (persistent) task group to all asyncio tasks 
spawned by a black-box code (when the black-box itself does not use task 
groups), we could achieve the full application-level transparency on the timing 
of task cancellation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

Joongi Kim  added the comment:

The main benefit is that any legacy code that I cannot modify can be upgraded 
to TaskGroup-based codes, which offers a better machinary for exception 
handling and propagation.

There may be different ways to visit this issue: allow replacing the task 
factory in asyncio at runtime.  Then I could just implement my own snippet to 
transfer the "ownership" of the task to a specific task group.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Yury Selivanov]

Yury Selivanov  added the comment:

> I believe that this approach would allow more control over tasks implicitly 
> spawned by 3rd-party libraries that cannot control.

Please elaborate. I'm not sure what are the benefits of this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Joongi Kim]

New submission from Joongi Kim :

Along with bpo-46843 and the new asyncio.TaskGroup API, I would like to suggest 
addition of context-based TaskGroup feature.

Currently asyncio.create_task() just creates a new task directly attached to 
the event loop, while asyncio.TaskGroup.create_task() creates a new task 
managed by the TaskGroup instance.

It would be ideal to all existing asyncio codes to migrate to use TaskGroup, 
but this is impractical.

An alternative approach is to implicitly bind asyncio.create_task() under a 
specific context to a specific task group, probably using contextvars.

I believe that this approach would allow more control over tasks implicitly 
spawned by 3rd-party libraries that cannot control.

How about your thoughts?

--
components: asyncio
messages: 413881
nosy: achimnol, asvetlov, gvanrossum, yselivanov
priority: normal
severity: normal
status: open
title: Context-based TaskGroup for legacy libraries
type: enhancement
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com