Thanks for the suggestion.
I'm not a fan of adding a layer that tries to be this clever. How would
possible prefetches be identified? What happens when an initial loop in a
view requires one prefetch, but a subsequent loop in a template requires
some other prefetch? What about nested
I'm in favour of *some* automated way of handling these prefetches, whether
it's opt-in or opt-out there should be some mechanism for protection.
Preferably with loud logging that directs users to the source of the
automated hand-holding so they have an opportunity to disable or fine tune
I agree with Marc here that the proposed optimizations are 'magical'. I
think when it comes to optimizations like these you simply cannot know
in advance whether doing extra queries is going to a be an optimization
or a pessimization. If I can come up with a single example where it
I'm biased here, Gordon was my boss for nearly three years, 2014-2016.
I'm in favour of adding the auto-prefetching, I've seen it work. It was
created around midway through last year and applied to our application's
Admin interface, which was covered in tests with *django-perf-rec*. After
> I agree with Marc here that the proposed optimizations are 'magical'. I
> think when it comes to optimizations like these you simply cannot know in
> advance whether doing extra queries is going to a be an optimization or a
I think Django automatically fetching data from the
In my current version each object keeps a reference to a WeakSet of the
results of the queryset it came from.
This is populated in _fetch_all and if it is populated then
ForwardManyToOneDescriptor does a prefetch across all the objects in the
WeakSet instead of it's regular fetching.
On Tue, Aug
I didn't answer your questions directly. Sorry for the quoting but it's the
easiest way to deal with a half dozen questions.
> How would possible prefetches be identified?
Wherever we currently automatically fetch a foreign key value.
> What happens when an initial loop in a view requires one
The warnings you propose would certainly be an improvement on the status
However for that to be a complete solution Django would also need to detect
places where there are redundant prefetch_relateds.
Additionally tools like the Admin and DRF would need to provide adequate
> Adam/Gordon, I'm interested in hearing how these changes led you to
> discovering stale prefetches?
We removed all the manual prefetches in admin get_queryset() methods, added
the auto_prefetch_related call, and regenerated the performance records
from django-perf-rec tests that hit the
The next() method is gone from the return value of open() in Python 3 and I
don't think it was every necessary. As per the iterator protcool (
https://docs.python.org/2/library/stdtypes.html#iterator-types ), __iter__
is meant to return an iterator over the object, and *that* is what the
Exploding query counts are definitely a pain point in Django, anything to
improve that is definitely worth considering. They have been a problem in
all Django projects I have seen.
However I think the correct solution is for developers to correctly add
select/prefetch calls. There is no general
I wonder if a solution similar to  from the rails world would satisfy
this request. Rather then doing anything 'magical' we instead log when we
detect things like accessing a related model that has not been pre-fetched.
On Tue, Aug 15, 2017 at 5:14 PM,
Sorry maybe I wasn't clear enough about the proposed mechanism.
Currently when you dereference a foreign key field on an object (so
'choice.question' in the examples above) if it doesn't have the value
cached from an earlier access, prefetch_related or select_related then
How is it implemented? Does each object keep a reference to the queryset it
On Tue, Aug 15, 2017 at 2:44 PM, Gordon Wrigley
> Sorry maybe I wasn't clear enough about the proposed mechanism.
> Currently when you dereference a
Automatically prefetching is something I feel should be avoided.
A common gripe I have with ORMs is they hide what's actually happening with
the database, resulting in beginners-going-on-intermediates building
libraries/systems that don't scale well.
We have several views in a dashboard, where a
I would rather have warnings as well, adding more magical behavior is bad
and might even degrade performance on some cases, automatically selecting a
bunch of data that "might" be used is bad, and specially considering how
slow python is, accidentally loading/building 1k+ objects when maybe
I think this is an excellent suggestion.
It seems generally accepted in this thread that although there are cases
where this would hurt performance, it would on average solve more problems
than it creates. The debate seems to be more whether or not it's "right"
for the ORM to behave in this
I believe we should be optimising for the **common** use case, not
expecting everyone to be experts with the ORM.
> If I can come up with a single example where it would significantly
decrease performance (either memory usage or speed) compared to the default
(and I'm sure I can), then I would
On Thu, Aug 10, 2017, michael.sch...@rz.uni-augsburg.de
>Is there a planned release date in the near future for django 1.11 support?
Some time soon - we can't give an exact date, but we are working on it.
The 2 goals of a famework:
- protect you from the tedious things
- protect you from the dangerous things
N+1 queries would be in the 'dangerous' category, IMHO, and 'detecting
causes of N+1 queries' is in the 'tedious'.
If we can at least add some DEBUG flagged machinery to detect and warn
I'd like to discuss automatic prefetching in querysets. Specifically
automatically doing prefetch_related where needed without the user having
to request it.
For context consider these three snippets using the Question & Choice
models from the tutorial
Mail list logo