> On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon.wrig...@gmail.com> wrote:
> I'd like to discuss automatic prefetching in querysets. Specifically 
> automatically doing prefetch_related where needed without the user having to 
> request it.


I'm rather sympathetic to this proposal. Figuring out N + 1 problems in the 
admin or elsewhere gets old.

In addition to everything that was said already, I'd like to point out that 
Django already has a very similar "magic auto prefetching" behavior in some 
cases :-)

I'm referring to the admin which calls select_related() on non-nullable foreign 
keys in the changelist view. The "non-nullable" condition makes that behavior 
hard to predict — I'd go as far as to call it non deterministic. For details, 
see slide 54 of 
 and the audio commentary at https://youtu.be/5fheDDj3oHY?t=2024 

The feature proposed here is most useful if it's opt-out because it targets 
people who aren't aware that the problem even exists — at best they notice that 
Django is slow and that reminds them vaguely of a rant that explains why ORMs 
are the worst thing since object oriented programming.

It should kick in only when no select_related or prefetch_related is in effect, 
to avoid interfering with pre-existing optimizations. It's still easy to 
construct an example where it would degrade performance but I don't think such 
situations will be common in practice. Still, there should be a per-queryset 
opt-out for these cases.

We may want to introduce it with a deprecation path, that is, make it opt-in at 
first and log a deprecation warning where the behavior would kick-in, so 
developers who want to disable it can add the per-queryset opt-out.

At this point, my main concerns are:

1) The difficulty of identifying where the queryset originates, given that 
querysets are lazy. Passing objects around is common; sometimes it can be hard 
to figure out where an object comes from. It isn't visible in the stack trace. 
In my opinion this is the strongest argument against the feature.

2) The lack of this feature for reverse one-to-one relations; it's only 
implemented for foreign keys. It's hard to tell them apart in Python code. The 
subtle differences, like return None vs. raise ObjectDoesNotExist when there's 
no related object, degrade the developer experience.

3) The strong opinions expressed against the feature. I'm not sure that 
consensus is within reach. If we can't agree that this is an adequate amount of 
magic, we're likely to stick with the status quo. I'd rather not have this 
question decided by a vote of the technical board.

In the grand scheme of things, going from "prefetching a related instance for 
an object" to "prefetching related instances for all objects in the queryset" 
isn't that much of a stretch... But I admit it's rather scary to make this 
change for all existing Django projects!

Best regards,


You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
For more options, visit https://groups.google.com/d/optout.
              • ... Gordon Wrigley
  • Re: Au... Cristiano Coelho
    • R... Alexander Hill
      • ... Josh Smeaton
        • ... Adam Johnson
          • ... 'Tom Evans' via Django developers (Contributions to Django itself)
            • ... Shai Berger
              • ... Josh Smeaton
              • ... Gordon Wrigley
        • ... Anssi Kääriäinen
  • Re: Au... Aymeric Augustin
    • R... Gordon Wrigley
    • R... Malcolm Box
      • ... Andrew Godwin
        • ... Collin Anderson
    • R... Aymeric Augustin
      • ... Anssi Kääriäinen
        • ... Shai Berger
          • ... Collin Anderson
          • ... Luke Plant
            • ... Tobias McNulty

Reply via email to