#35238: Missing chunk_size kwarg throws exception when creating test DB if model
has base-manager with prefetches
               Reporter:  alexcleduc            |          Owner:  nobody
                   Type:  Bug                   |         Status:  new
              Component:  Core (Serialization)  |        Version:  5.0
               Severity:  Normal                |       Keywords:
           Triage Stage:  Unreviewed            |      Has patch:  1
    Needs documentation:  0                     |    Needs tests:  0
Patch needs improvement:  0                     |  Easy pickings:  0
                  UI/UX:  0                     |
 I am trying to upgrade from 4.x to 5.0 and am running into this issue when
 creating my test database.

 I'm using pytest, but I'd be very surprised if this wasn't also an issue
 with unit-test. That said, I do find it a little surprising no one has
 noticed this. Surely there would have been other projects out with models
 whose base-managers have prefetches? So I'm not ''entirely'' confident
 this is a bug in django. Maybe pytest should be initializing test DBs
 differently, I haven't done that research.

 See django/db/backends/base/creation.py for the offending code:

 class BaseDatabaseCreation:
    # ...
     def serialize_db_to_string(self):
         # ...
         # Iteratively return every object for all models to serialize.
         def get_objects():
             for app_config in apps.get_app_configs():
                 if (...):
                     for model in app_config.get_models():
                         if (...):
                             queryset = model._base_manager.using(
                             yield from queryset.iterator() # HERE THROWS


 Here is the call-stack in my project, you'll see that although it starts
 with pytest, there's a chain of 6 first-party django calls that lead to
 the exception:


   File "/.../site-packages/pytest_django/fixtures.py", line 139, in
     db_cfg = setup_databases(
   File ".../site-packages/django/test/utils.py", line 203, in
   File ".../site-packages/django/db/backends/base/creation.py", line 94,
 in create_test_db
     self.connection._test_serialized_contents =
   File ".../site-packages/django/db/backends/base/creation.py", line 142,
 in serialize_db_to_string
     serializers.serialize("json", get_objects(), indent=None, stream=out)
   File ".../site-packages/django/core/serializers/__init__.py", line 134,
 in serialize
     s.serialize(queryset, **options)
   File ".../site-packages/django/core/serializers/base.py", line 108, in
     for count, obj in enumerate(queryset, start=1):
   File ".../site-packages/django/db/backends/base/creation.py", line 138,
 in get_objects
     yield from queryset.iterator()

 **Proposed solution**
 I am putting up a PR that will conditionally use the chunk_size kwarg when
 calling queryset.iterator() in django/db/backends/base/creation.py. An
 alternative is to set a fallback chunk_size in iterator() itself, if
 prefetches are found, but I am guessing there's a good reason that wasn't
 done in the first place. I am defaulting to the same as aiterator, which
 is 2000

 I am not confident in this solution, there seem to be quite a bit of
 first-party calls to queryset.iterator that don't include a chunk-size
 kwarg, however this fix makes my project's relatively large test-suite
 pass, and I have prefetches on many models' base-managers including the
 user model.
Ticket URL: <https://code.djangoproject.com/ticket/35238>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Reply via email to