Testing Unmanaged Models - Using the SchemaEditor to create db tables

[Emmanuel Katchy]

Hi everyone!

I'd like to get your thoughts on something.

Unmanaged models mean that Django no longer handles creating and managing 
schema at the database level (hence the name).
When running tests, this means these tables aren't created, and we can't 
run queries against that model. The general solution I found is to monkey-patch 
the TestSuiteRunner to temporarily treat models as managed 

.

Doing a bit of research I however came up with a solution using SchemaEditor 
, to create the 
model tables directly, viz:

```
"""
A cleaner approach to temporarily creating unmanaged model db tables for 
tests
"""

from unittest import TestCase

from django.db import connections, models

class create_unmanaged_model_tables:
"""
Create db tables for unmanaged models for tests
Adapted from: https://stackoverflow.com/a/49800437
Examples:
with create_unmanaged_model_tables(UnmanagedModel):
...
@create_unmanaged_model_tables(UnmanagedModel, FooModel)
def test_generate_data():
...

@create_unmanaged_model_tables(UnmanagedModel, FooModel)
def MyTestCase(unittest.TestCase):
...
"""

def __init__(self, unmanaged_models: list[ModelBase], db_alias: str = 
"default"):
"""
:param str db_alias: Name of the database to connect to, defaults 
to "default"
"""
self.unmanaged_models = unmanaged_models
self.db_alias = db_alias
self.connection = connections[db_alias]

def __call__(self, obj):
if issubclass(obj, TestCase):
return self.decorate_class(obj)
return self.decorate_callable(obj)

def __enter__(self):
self.start()

def __exit__(self, exc_type, exc_value, traceback):
self.stop()

def start(self):
with self.connection.schema_editor() as schema_editor:
for model in self.unmanaged_models:
schema_editor.create_model(model)

if (
model._meta.db_table
not in self.connection.introspection.table_names()
):
raise ValueError(
"Table `{table_name}` is missing in test 
database.".format(
table_name=model._meta.db_table
)
)

def stop(self):
with self.connection.schema_editor() as schema_editor:
for model in self.unmanaged_models:
schema_editor.delete_model(model)

def copy(self):
return self.__class__(
unmanaged_models=self.unmanaged_models, db_alias=self.db_alias
)

def decorate_class(self, klass):
# Modify setUpClass and tearDownClass
orig_setUpClass = klass.setUpClass
orig_tearDownClass = klass.tearDownClass

@classmethod
def setUpClass(cls):
self.start()
if orig_setUpClass is not None:
orig_setUpClass()


@classmethod
def tearDownClass(cls):
if orig_tearDownClass is not None:
orig_tearDownClass()
self.stop()

klass.setUpClass = setUpClass
klass.tearDownClass = tearDownClass

return klass

def decorate_callable(self, callable_obj):
@functools.wraps(callable_obj)
def wrapper(*args, **kwargs):
with self.copy():
return callable_obj(*args, **kwargs)

return wrapper
```

Would this make a good addition to *django.test.utils*?

P.S: First time posting here :P


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6f5c668b-8f72-42be-9e41-01c786c12027n%40googlegroups.com.

Possible async view regression in django 5.0?

[Hugo Heyman]

Hello!

I've been troubleshooting an issue with lingering idle db connections 
(postgres) since upgrading from django 4.2 to 5.0. The issue manifested by 
db raising OperationalError FATAL: sorry, too many clients already not long 
after deploying the new version. We're running asgi with uvicorn + gunicorn.

After extensive troubleshooting I installed django from a local repo and 
ran a git bisect to pin down the commit where this first occurred and ended 
up on this commit 
https://github.com/django/django/commit/64cea1e48f285ea2162c669208d95188b32bbc82

Since I've recently built something quite async heavy I decided to dig into 
how the ASGIHandler code works and see if I can understand it and possibly 
make a fix if there really is some issue there.

Now for the possible regression with django:

The change introduced in Django 5.0 to allow handling of 
asyncio.CancelledError in views seems to also allow async views with normal 
HttpResponse handling client disconnects. As a consequence though the 
request_finished signal may not run leading to db connection not being 
closed.

What currently seems to be happening in ASGIHandler in 
django/core/handlers/asgi.py goes something like this:

1. Concurrent tasks are started for
  - disconnect listener (listen for "http.disconnect" from webserver)
  - process_request
2. If disconnect happens first (e.g. browser refresh) cancel() is run on 
process_request task and request_finished may not have been triggered
3. db connection for the request is not closed :(

Here are the lines of code where all this happens, marked: 
https://github.com/django/django/blob/9c6d7b4a678b7bbc6a1a14420f686162ba9016f5/django/core/handlers/asgi.py#L191-L232

Possible fix? I'm thinking only views that return a HTTPStreamingResponse 
would need to allow for cancellation cleanup handling via the view. If so 
response = await self.run_get_response(request) could run before and 
outside of any task so we could check response.streaming attribute and only 
run listen_for_disconnect task when there's a streaming response.

I've made a patch to try the above out and it seems to fix the issues. 
https://github.com/HeyHugo/django/commit/e25a1525654e00dcd5b483689ef16e0dc74d32d1

Here's a minimal setup to demonstrate the issue via print debugging: 
https://github.com/HeyHugo/django_async_issue

(I have never found a real bug in django so I'm still having doubts :D)

Best regards
/Hugo

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4c7a761b-1420-4997-900f-b8f0d59f4f31n%40googlegroups.com.