#36714: Async signals lose ContextVar state due to use of asyncio.gather
-------------------------------------+-------------------------------------
Reporter: Mykhailo Havelia | Type:
| Uncategorized
Status: new | Component: HTTP
| handling
Version: dev | Severity: Normal
Keywords: asyncio, signals | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
The natural way to share global, per-request state in asyncio is through
contextvars. In Django, this is typically used via `asgiref.local.Local`.
However, Django's async signal dispatch currently uses `asyncio.gather`,
which internally creates new tasks (`asyncio.create_task`). This breaks
context propagation, since each task gets its own copy of the context. As
a result, it's impossible to set a global (context-based) variable inside
a signal handler and have it shared with other signal handlers or parts of
the same request/response cycle.
Example
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
request_id = contextvars.ContextVar('request_id', default=None)
async def set_global_variable(*args, **kwargs):
# set global variable
request_id.set('request_id_value')
print('get value', request_id.get())
signals.request_started.connect(set_global_variable)
async def index(request: HttpRequest) -> HttpResponse:
# get global variable
print('request_id', request_id.get())
return HttpResponse(content=request_id.get())
urlpatterns = [path("", index), ]
}}}
result
{{{
get value request_id_value
request_id None
}}}
The value set inside the signal handler is lost, because the handler runs
in a separate task with its own context.
If we are talking exactly about `signals.request_started` and
`signals.request_finished`, they are typically used for setting up and
cleaning up per-request resources. With `asyncio.gather`, cleanup logic
that relies on `ContextVar` cannot work properly.
{{{
from django.core import signals
from django.http import (
HttpRequest,
HttpResponse,
)
import contextvars
from django.urls import path
db_connection = contextvars.ContextVar('db_connection', default=None)
async def get_or_create_connection():
if not db_connection.get():
db_connection.set('connection')
return db_connection.get()
async def close_connection(*args, **kwargs):
connection = db_connection.get()
if not connection:
print('cannot clean - connection does not exist')
return
print('close connection')
connection.set(None)
signals.request_finished.connect(close_connection)
async def index(request: HttpRequest) -> HttpResponse:
# create connection inside handler
connection = await get_or_create_connection()
# await get_data(connection)
return HttpResponse(content="ok")
urlpatterns = [path("", index), ]
}}}
result
{{{
cannot clean - connection does not exist
}}}
**Expected behavior**
Signal handlers should run in the same async context as the request,
preserving `ContextVar` and `asgiref.local.Local` state.
**Proposed solution**
Signal:
Dispatch async signal handlers sequentially (or via direct await) instead
of using `asyncio.gather`, so that the existing execution context is
preserved throughout the request lifecycle. Yes, this change removes
parallelism, but that shouldn’t be a major concern. The only real benefit
of running signal handlers in parallel would be for IO-bound operations -
yet in most cases, these handlers interact with the same database
connection. Since database operations aren’t truly parallel under the
hood, the performance gain from `asyncio.gather` is negligible.
ASGIHandler:
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send)),
]
...
await signals.request_finished.asend(sender=self.__class__)
}}}
Global variables created inside `process_request` are not visible to
`request_finished`, because each task runs in a separate context. We can
try using `contextvars.copy_context()` to preserve and share the same
context between tasks and signal handlers.
{{{
async def handle(self, scope, receive, send):
...
await signals.request_started.asend(sender=self.__class__,
scope=scope)
ctx = contextvars.copy_context()
tasks = [
asyncio.create_task(self.listen_for_disconnect(receive)),
asyncio.create_task(process_request(request, send), context=ctx),
]
...
await
asyncio.create_task(signals.request_finished.asend(sender=self.__class__),
context=ctx)
}}}
Here is a simple example
{{{
import asyncio
import contextvars
global_state = contextvars.ContextVar('stage', default=0)
async def inc():
value = global_state.get()
print('value: ', value)
global_state.set(value + 1)
async def main():
await asyncio.create_task(inc())
await asyncio.create_task(inc())
await asyncio.create_task(inc())
print('first: ', global_state.get())
ctx = contextvars.copy_context()
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
await asyncio.create_task(inc(), context=ctx)
print('second: ', ctx.get(global_state))
await main()
}}}
result
{{{
value: 0
value: 0
value: 0
first: 0
value: 0
value: 1
value: 2
second: 3
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36714>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019a58cdb028-93498b74-f1bb-4a96-a2a6-9ceb00516bd8-000000%40eu-central-1.amazonses.com.
