> On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon.wrig...@gmail.com> wrote: > > I'd like to discuss automatic prefetching in querysets. Specifically > automatically doing prefetch_related where needed without the user having to > request it.
Hello, I'm rather sympathetic to this proposal. Figuring out N + 1 problems in the admin or elsewhere gets old. In addition to everything that was said already, I'd like to point out that Django already has a very similar "magic auto prefetching" behavior in some cases :-) I'm referring to the admin which calls select_related() on non-nullable foreign keys in the changelist view. The "non-nullable" condition makes that behavior hard to predict — I'd go as far as to call it non deterministic. For details, see slide 54 of https://myks.org/data/20161103-Django_Under_the_Hood-Debugging_Performance.pdf <https://myks.org/data/20161103-Django_Under_the_Hood-Debugging_Performance.pdf> and the audio commentary at https://youtu.be/5fheDDj3oHY?t=2024 <https://youtu.be/5fheDDj3oHY?t=2024>. The feature proposed here is most useful if it's opt-out because it targets people who aren't aware that the problem even exists — at best they notice that Django is slow and that reminds them vaguely of a rant that explains why ORMs are the worst thing since object oriented programming. It should kick in only when no select_related or prefetch_related is in effect, to avoid interfering with pre-existing optimizations. It's still easy to construct an example where it would degrade performance but I don't think such situations will be common in practice. Still, there should be a per-queryset opt-out for these cases. We may want to introduce it with a deprecation path, that is, make it opt-in at first and log a deprecation warning where the behavior would kick-in, so developers who want to disable it can add the per-queryset opt-out. At this point, my main concerns are: 1) The difficulty of identifying where the queryset originates, given that querysets are lazy. Passing objects around is common; sometimes it can be hard to figure out where an object comes from. It isn't visible in the stack trace. In my opinion this is the strongest argument against the feature. 2) The lack of this feature for reverse one-to-one relations; it's only implemented for foreign keys. It's hard to tell them apart in Python code. The subtle differences, like return None vs. raise ObjectDoesNotExist when there's no related object, degrade the developer experience. 3) The strong opinions expressed against the feature. I'm not sure that consensus is within reach. If we can't agree that this is an adequate amount of magic, we're likely to stick with the status quo. I'd rather not have this question decided by a vote of the technical board. In the grand scheme of things, going from "prefetching a related instance for an object" to "prefetching related instances for all objects in the queryset" isn't that much of a stretch... But I admit it's rather scary to make this change for all existing Django projects! Best regards, -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/2D9DCC0F-EAB0-4512-8AEC-08A694DF9074%40polytechnique.org. For more options, visit https://groups.google.com/d/optout.
