Re: Automatic prefetching in querysets

Hi Gordon, Thanks for the suggestion. I'm not a fan of adding a layer that tries to be this clever. How would possible prefetches be identified? What happens when an initial loop in a view requires one prefetch, but a subsequent loop in a template requires some other prefetch? What about nested

Re: Automatic prefetching in querysets

I'm in favour of *some* automated way of handling these prefetches, whether it's opt-in or opt-out there should be some mechanism for protection. Preferably with loud logging that directs users to the source of the automated hand-holding so they have an opportunity to disable or fine tune the

Re: Automatic prefetching in querysets

I agree with Marc here that the proposed optimizations are 'magical'. I think when it comes to optimizations like these you simply cannot know in advance whether doing extra queries is going to a be an optimization or a pessimization. If I can come up with a single example where it would

Re: Automatic prefetching in querysets

I'm biased here, Gordon was my boss for nearly three years, 2014-2016. I'm in favour of adding the auto-prefetching, I've seen it work. It was created around midway through last year and applied to our application's Admin interface, which was covered in tests with *django-perf-rec*. After adding

Re: Automatic prefetching in querysets

> > I agree with Marc here that the proposed optimizations are 'magical'. I > think when it comes to optimizations like these you simply cannot know in > advance whether doing extra queries is going to a be an optimization or a > pessimization. I think Django automatically fetching data from the

Re: Automatic prefetching in querysets

In my current version each object keeps a reference to a WeakSet of the results of the queryset it came from. This is populated in _fetch_all and if it is populated then ForwardManyToOneDescriptor does a prefetch across all the objects in the WeakSet instead of it's regular fetching. On Tue, Aug

Re: Automatic prefetching in querysets

I didn't answer your questions directly. Sorry for the quoting but it's the easiest way to deal with a half dozen questions. > How would possible prefetches be identified? Wherever we currently automatically fetch a foreign key value. > What happens when an initial loop in a view requires one

Re: Automatic prefetching in querysets

The warnings you propose would certainly be an improvement on the status quo. However for that to be a complete solution Django would also need to detect places where there are redundant prefetch_relateds. Additionally tools like the Admin and DRF would need to provide adequate hooks for

Re: Automatic prefetching in querysets

> > Adam/Gordon, I'm interested in hearing how these changes led you to > discovering stale prefetches? We removed all the manual prefetches in admin get_queryset() methods, added the auto_prefetch_related call, and regenerated the performance records from django-perf-rec tests that hit the

Re: Should django File wrapper support .next()?

The next() method is gone from the return value of open() in Python 3 and I don't think it was every necessary. As per the iterator protcool ( https://docs.python.org/2/library/stdtypes.html#iterator-types ), __iter__ is meant to return an iterator over the object, and *that* is what the next()

Re: Automatic prefetching in querysets

Exploding query counts are definitely a pain point in Django, anything to improve that is definitely worth considering. They have been a problem in all Django projects I have seen. However I think the correct solution is for developers to correctly add select/prefetch calls. There is no general

Re: Automatic prefetching in querysets

I wonder if a solution similar to [1] from the rails world would satisfy this request. Rather then doing anything 'magical' we instead log when we detect things like accessing a related model that has not been pre-fetched. [1] https://github.com/flyerhzm/bullet On Tue, Aug 15, 2017 at 5:14 PM,

Re: Automatic prefetching in querysets

Sorry maybe I wasn't clear enough about the proposed mechanism. Currently when you dereference a foreign key field on an object (so 'choice.question' in the examples above) if it doesn't have the value cached from an earlier access, prefetch_related or select_related then Django will

Re: Automatic prefetching in querysets

Hi Gordon, How is it implemented? Does each object keep a reference to the queryset it came from? Collin On Tue, Aug 15, 2017 at 2:44 PM, Gordon Wrigley wrote: > Sorry maybe I wasn't clear enough about the proposed mechanism. > > Currently when you dereference a

Re: Automatic prefetching in querysets

Automatically prefetching is something I feel should be avoided. A common gripe I have with ORMs is they hide what's actually happening with the database, resulting in beginners-going-on-intermediates building libraries/systems that don't scale well. We have several views in a dashboard, where a

Re: Automatic prefetching in querysets

I would rather have warnings as well, adding more magical behavior is bad and might even degrade performance on some cases, automatically selecting a bunch of data that "might" be used is bad, and specially considering how slow python is, accidentally loading/building 1k+ objects when maybe

Re: Automatic prefetching in querysets

I think this is an excellent suggestion. It seems generally accepted in this thread that although there are cases where this would hurt performance, it would on average solve more problems than it creates. The debate seems to be more whether or not it's "right" for the ORM to behave in this

Re: Aldryn Newsblog and Django 1.10 (or 1.11 LTS) Support

On Thu, Aug 10, 2017, michael.sch...@rz.uni-augsburg.de wrote: >Is there a planned release date in the near future for django 1.11 support? Some time soon - we can't give an exact date, but we are working on it. Daniele -- Message URL:

Automatic prefetching in querysets

I'd like to discuss automatic prefetching in querysets. Specifically automatically doing prefetch_related where needed without the user having to request it. For context consider these three snippets using the Question & Choice models from the tutorial