#22757: prefetch_related isn't as effecient as it could be with
GenericForeignKey
and proxy models
-------------------------------------+-------------------------------------
Reporter: gwahl@… | Owner: (none)
Type: | Status: new
Cleanup/optimization |
Component: Database layer | Version: dev
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch_related | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
Thanks for having a shot at this Nolan!
I think that a lot of the complexity here stems from the fact doesn't
support polymorphic querysets that is querysets that can return multiple
models.
This poses a challenge from an implementation perspective for this issue
because the (simplified here) return signature of `get_prefetch_querysets`
contains three members
1. A collection of prefetch querysets
2. A callable that takes a model instance returned from one of the
prefetch queries and return a tuple uniquely identifying the item (that's
`lambda obj: (obj.pk, obj.__class__)`)
3. A callable that take model instances related objects are prefetched for
and return a tuple uniquely identifying the related items (that's
`gfk_key`)
The `prefetch_related` logic then executes all the querysets from 1.,
build a map from 2, and inserts it with 3. to do the proper
`_prefetch_related_cache` assignments.
Since Django only allows for a single model class to be used to construct
objects returned from a queryset, remember the lack of polymorphism
support, then one might be tempted to annotate objects with a
`_content_type_id` attribute to have the logic in 2. changed to `lambda
obj: (obj.pk, self.get_content_type(id=obj._content_type_id,
using=obj._state.db).model_class()` so the proper mapping occurs. Doing do
doesn't solve the problem of proper `__class__` assignment as you've
brought up Nolan nor does it address another edge case that would have
probably caused during review.
Even if we're able to return a specialized `Queryset` that overrides the
`_iterable_class` to return mixed models ([https://github.com/django-
polymorphic/django-
polymorphic/blob/5111fb070e4e0b0153e816d163871f2039ca2ae2/polymorphic/query.py#L25
it's the tactic used by packages like django-polymorphic] for example) how
we deal with the case where the same id is needs to be prefetched both
concretely and for it's proxy.
In other words, given the reported scenario, what kind of SQL query would
we generate for
{{{#!python
child = Child.objects.create(id=1)
proxy_child = ProxiedChild.objects.get(id=1)
parent = Parent.objects.create(child=child)
proxy_parent = Parent.objects.create(child=proxy_child)
Parent.objects.prefetch_related("child")
}}}
In this case the same `foo_child` row needs to returned twice one as a
child and one as a proxy_child which cannot be achieved with a simple `id
IN (1, 1)`.
The only way I can think of dealing with this edge case is by using a
`UNION ALL` with combined annotation so the following SQL would be
generated
{{{#!sql
-- Assuming the content type of the Child model is 2 and the content type
of the ProxiedChild model is 3
SELECT "foo_child"."id", 2 AS "_content_type_id" FROM "foo_child" WHERE
"foo_child"."id" IN (1)
UNION ALL
SELECT "foo_child"."id", 3 AS "_content_type_id" FROM "foo_child" WHERE
"foo_child"."id" IN (1)
}}}
And that must then be combined with the `_iterable_class` override
mentioned above.
There's even more fun now that we've added `GenericPrefetch` in 5.0 as it
allows you to specify querysets for each of the models involved which
means that it allows passing querysets with `select_related` and
annotations which cannot use the `UNION ALL` approach described above so
the ''optimization'' would have to be disabled.
After looking at this into more details I also agree that the addition in
complexity to avoid this query is likely not worth it as it would either
require some potentially expensive Python workaround or significant boiler
plate to get working properly. I suspect we'll want to ''wontfix'' this
one unfortunately.
--
Ticket URL: <https://code.djangoproject.com/ticket/22757#comment:7>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/0107018b98026ec4-6a9458a6-3d4a-4cef-8f3a-bb8fc2d220d7-000000%40eu-central-1.amazonses.com.
