#26565: Allow Prefetch query to use .values()
-------------------------------------+-------------------------------------
Reporter: Maxime Lorant | Owner: nobody
Type: New feature | Status: new
Component: Database layer | Version: master
(models, ORM) |
Severity: Normal | Resolution:
Keywords: prefetch, values | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Micah Lyle):
I took a stab at this during the DjangoCon 2019 sprints.
I had very limited experience with the prefetching/ORM internals before
this, so there could be workarounds/solutions that I'm unaware of, but
here's what I found so far.
One thing I tried to tackle is handling the many to many case with just
`values()`. My goal was to take the `'_prefetch_related_val_%s'` in the
`queryset.extra(...)` call in the `get_prefetch_queryset()`
`ManyRelatedManager` and add/apply it to whatever `values()` call was made
and then join that by doing `pop()` from the dictionary instead of a
`getattr` call (which would effectively give the developer the same query
that they originally made by modifying it under the hood). I ran into some
issues doing this though:
"Any `extra()` call made after a `values()` call will have its extra
selected fields ignored" (from the Django 2.2 docs)."
This means that the `queryset.extra(...)` call in the `ManyRelatedManager`
(which from my quick search appears to be the only remaining usage of
`extra(...)` across the entire Django codebase (except the tests)) is
effectively ignored, and even if I try to add a `.values(*existing_values,
'_prefetch_related_val_%s')` call, it returns this error (from my example
test/use case using the models from the prefetch tests):
{{{
queryset.values('id', 'title', '_prefetch_related_val_author_id')
django.core.exceptions.FieldError: Cannot resolve keyword
'_prefetch_related_val_author_id' into field. Choices are:
_prefetch_related_val_author_id, authors, bio, bookwithyear,
first_time_authors, id, read_by, title
}}}
Is this a bug? It says it can't resolve a field and then says that field
is a choice.
I thought about trying to workaround it by essentially "undoing" the
`values()` part of the `queryset` and then reapplying it at the end with
the value from the join table (which then gets `pop`ed, but I'm not sure
if that's feasible/doable and what that would actually look like. Besides,
this entire solution so far feels somewhat hacky in the first place.
I think that whatever change makes this happen may require some sort of
specification on the developer's end for how to apply the join from the
prefetched dictionaries to the existing list of instances. Maybe even
writing parts of a custom prefetcher and specifying that custom prefetcher
as a fourth keyword argument (something I explored, but didn't get very
far) to the `Prefetch` object, and then tweaking the logic in
`prefetch_related_objects` and other functions to support using that
fourth keyword argument.
I'm also wondering if there's a way with Django 3.1+ technology/API to
rewrite the `queryset.extra(...)` call in `ManyRelatedManager`. An out
there idea I had was replacing the `queryset.extra(...)` call with a query
on the through table and then doing a select related to the foreign key
that is on the side we want to bring in to join to the existing instances
(in the case above, we'd grab
`Book.authors.through.objects.filter_based_on_what_we_have_for_books_and_authors().select_related('book')`
and then transform to a values result from there but that would (I think)
prevent aggregations in the first place and run into a number of other
problems.
--
Ticket URL: <https://code.djangoproject.com/ticket/26565#comment:14>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/065.bc27cd3ad77e6c908b55310b13b74907%40djangoproject.com.
