Re: Automatic prefetching in querysets

2017-09-12 Thread Gordon Wrigley
com> wrote: > I'd like to discuss automatic prefetching in querysets. Specifically > automatically doing prefetch_related where needed without the user having > to request it. > > For context consider these three snippets using the Question & Choice > model

Re: Automatic prefetching in querysets

2017-08-30 Thread Patryk Zawadzki
W dniu środa, 16 sierpnia 2017 14:48:54 UTC+2 użytkownik Luke Plant napisał: > > I completely agree that visibility of this problem is a major issue, and > would really welcome work on improving this, especially in DEBUG mode. One > option might be a method that replaces lazy loads with

Re: Automatic prefetching in querysets

2017-08-21 Thread Shai Berger
On Monday 21 August 2017 18:44:35 Tobias McNulty wrote: > On Sat, Aug 19, 2017 at 1:10 PM, Luke Plant wrote: > > This could work something like the way that ForeignKey `on_delete` works > > - you have options that are enumerated as constants, but in reality they > > are

Re: Automatic prefetching in querysets

2017-08-21 Thread Tobias McNulty
On Sat, Aug 19, 2017 at 1:10 PM, Luke Plant wrote: > This could work something like the way that ForeignKey `on_delete` works - > you have options that are enumerated as constants, but in reality they are > classes that embody the strategy to be used. We could have

Re: Automatic prefetching in querysets

2017-08-19 Thread Luke Plant
This could work something like the way that ForeignKey `on_delete` works - you have options that are enumerated as constants, but in reality they are classes that embody the strategy to be used. We could have something similar - `on_missing_relation=FETCH | WARN | ERROR | IGNORE ... ` Luke

Re: Automatic prefetching in querysets

2017-08-19 Thread Todor Velichkov
+1 to just add the queryset method it gives a fine granular level of control. Moreover when not used we could display warnings which can help people detect these O(n) query cases as Tom Forbes initially suggested. On Friday, August 18, 2017 at 9:08:11 AM UTC+3, Anssi Kääriäinen wrote: > > On

Re: Automatic prefetching in querysets

2017-08-18 Thread Collin Anderson
I like that idea - keep it a private API for now and make it a public API once people have used it a little bit. On Fri, Aug 18, 2017 at 4:01 AM, Shai Berger wrote: > On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote: > > Maybe we should just add the queryset method.

Re: Automatic prefetching in querysets

2017-08-18 Thread Shai Berger
On Friday 18 August 2017 09:08:11 Anssi Kääriäinen wrote: > Maybe we should just add the queryset method. This is the smallest atomic > task that can be done. Even if there's only the queryset method available, > it's possible to enable prefetches per model by using a Manager. > I disagree on

Re: Automatic prefetching in querysets

2017-08-18 Thread Anssi Kääriäinen
On Thursday, August 17, 2017 at 11:30:45 PM UTC+3, Aymeric Augustin wrote: > > Hello, > > > On 17 Aug 2017, at 14:48, Luke Plant > wrote: > > > > On 16/08/17 23:17, Aymeric Augustin wrote: > >> It should kick in only when no select_related or prefetch_related is in >

Re: Automatic prefetching in querysets

2017-08-17 Thread Aymeric Augustin
Hello, > On 17 Aug 2017, at 14:48, Luke Plant wrote: > > On 16/08/17 23:17, Aymeric Augustin wrote: >> It should kick in only when no select_related or prefetch_related is in >> effect, to avoid interfering with pre-existing optimizations. It's still >> easy to

Re: Automatic prefetching in querysets

2017-08-17 Thread Collin Anderson
the first time they see the warning / see a problem. >> Experts can choose how they migrate - as Adam points out, even experts can >> miss things. >> >> Finally after a suitable warning period, this can become an opt-out >> feature and we arrive in the su

Re: Automatic prefetching in querysets

2017-08-17 Thread Andrew Godwin
; > Cheers, > > Malcolm > > On Wednesday, 16 August 2017 21:17:49 UTC+1, Aymeric Augustin wrote: >> >> On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon@gmail.com> wrote: >> >> I'd like to discuss automatic prefetching in querysets. Specifically >&

Re: Automatic prefetching in querysets

2017-08-17 Thread Malcolm Box
er for all users. Cheers, Malcolm On Wednesday, 16 August 2017 21:17:49 UTC+1, Aymeric Augustin wrote: > > On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon@gmail.com > > wrote: > > I'd like to discuss automatic prefetching in querysets. Specifically > automatically d

Re: Automatic prefetching in querysets

2017-08-17 Thread Gordon Wrigley
I'm not advocating either way on this, but it's probably worth noting that the context manager proposal is compatible with a queryset level optin/out and an object level optout. So you could for example have prefetch_related(auto=None) which can be explicitly set to True / False and takes it's

Re: Automatic prefetching in querysets

2017-08-16 Thread Josh Smeaton
I think there's a lot right with your suggestions here Shai. It delivers better default behaviour for new projects, does not affect existing deployments, and seems pretty easy to enable/disable selectively at any level of the stack. My only concern would be libraries leaning on this behaviour

Re: Automatic prefetching in querysets

2017-08-16 Thread Shai Berger
First of all, I think making the auto-prefetch feature available in some form is a great idea. As an opt-in, I would have made use of it on several occasions, and cut days of optimization work to minutes. It's true that this wouldn't achieve the best possible optimization in many cases, but it

Re: Automatic prefetching in querysets

2017-08-16 Thread Gordon Wrigley
Regarding 2, it does work for reverse one-to-one relations. On Wed, Aug 16, 2017 at 9:17 PM, Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon.wrig...@gmail.com> wrote: > > I'd like to discuss automatic pre

Re: Automatic prefetching in querysets

2017-08-16 Thread Aymeric Augustin
> On 15 Aug 2017, at 11:44, Gordon Wrigley <gordon.wrig...@gmail.com> wrote: > > I'd like to discuss automatic prefetching in querysets. Specifically > automatically doing prefetch_related where needed without the user having to > request it. Hello, I'm rather sympath

Re: Automatic prefetching in querysets

2017-08-16 Thread 'Tom Evans' via Django developers (Contributions to Django itself)
Is this opt-{in,out} considered to be a global flag, meant to be toggled on or off depending on whether it is an "expert" working on the project or not? I don't think that would be a good idea, almost all of our projects have a mix of skill levels, and people move from team to team on a regular

Re: Automatic prefetching in querysets

2017-08-16 Thread Gordon Wrigley
or a >>>> pessimization. If I can come up with a single example where it would >>>> significantly decrease performance (either memory usage or speed) compared >>>> to the default (and I'm sure I can), then I would be strongly opposed to it >>>>

Re: Automatic prefetching in querysets

2017-08-16 Thread Tom Forbes
t;> >>> Hi Gordon, >>> >>> Thanks for the suggestion. >>> >>> I'm not a fan of adding a layer that tries to be this clever. How would >>> possible prefetches be identified? What happens when an initial loop in a >>> view requires one

Re: Automatic prefetching in querysets

2017-08-16 Thread Marc Tamlyn
es things >>> together. I can't imagine ever wanting to use it, though, I would always >>> prefer the manual option. >>> >>> Luke >>> >>> >>> >>> On 15/08/17 21:02, Marc Tamlyn wrote: >>> >>> Hi Gordon, >>&g

Re: Automatic prefetching in querysets

2017-08-16 Thread Luke Plant
they go about identifying the cases where it is necessary? On 15 August 2017 at 10:44, Gordon Wrigley <gordon@gmail.com > wrote: I'd like to discuss automatic prefetching in querysets. Specifically automatically doing prefetch_related wher

Re: Automatic prefetching in querysets

2017-08-16 Thread Anssi Kääriäinen
On Wednesday, August 16, 2017 at 2:27:11 PM UTC+3, Josh Smeaton wrote: > > It won't affect experienced users. They'll read the release notes, see >> that this change has been implemented, and either go and delete a bunch of >> prefetch_related() calls, grumble a bit and turn auto-prefetch off

Re: Automatic prefetching in querysets

2017-08-16 Thread Adam Johnson
> > I wouldn't want is to give free optimisations to non-ORM pros > Something like 99% of django projects are by non-ORM pros, it can be easy to forget that on a mailing list of experts. On 16 August 2017 at 12:27, Josh Smeaton wrote: > It won't affect experienced

Re: Automatic prefetching in querysets

2017-08-16 Thread Josh Smeaton
> > It won't affect experienced users. They'll read the release notes, see > that this change has been implemented, and either go and delete a bunch of > prefetch_related() calls, grumble a bit and turn auto-prefetch off globally > or just file it away as another fact they know about the

Re: Automatic prefetching in querysets

2017-08-15 Thread Alexander Hill
on, proper joins here and there, unique > constraints, locks and race conditions, someone attempting to code > something that's not a blog or hello world really needs to know a bit about > all of that. > > > > El martes, 15 de agosto de 2017, 6:44:19 (UTC-3), Gordon Wrig

Re: Automatic prefetching in querysets

2017-08-15 Thread Cristiano Coelho
onstraints, locks and race conditions, someone attempting to code something that's not a blog or hello world really needs to know a bit about all of that. El martes, 15 de agosto de 2017, 6:44:19 (UTC-3), Gordon Wrigley escribió: > > I'd like to discuss automatic prefetching in queryse

Re: Automatic prefetching in querysets

2017-08-15 Thread Curtis Maloney
Do you know of any other ORMs which attempt similar magical optimisations? How do they go about identifying the cases where it is necessary? On 15 August 2017 at 10:44, Gordon Wrigley <gordon@gmail.com > wrote: I'd like to discuss

Re: Automatic prefetching in querysets

2017-08-15 Thread Josh Smeaton
king with an ORM. >> >> Do you know of any other ORMs which attempt similar magical >> optimisations? How do they go about identifying the cases where it is >> necessary? >> >> On 15 August 2017 at 10:44, Gordon Wrigley <gordon@gmail.com >> > wr

Re: Automatic prefetching in querysets

2017-08-15 Thread Anthony King
imisations? > How do they go about identifying the cases where it is necessary? > > On 15 August 2017 at 10:44, Gordon Wrigley <gordon.wrig...@gmail.com> > wrote: > >> I'd like to discuss automatic prefetching in querysets. Specifically >> automatically doing prefet

Re: Automatic prefetching in querysets

2017-08-15 Thread Adam Johnson
one and often (due to nulls and duplicates) >>>>> smaller. >>>>> >>>>> I do not propose touching reverse foreign key or many2many fields as >>>>> the generated queries could request substantially more rows from the DB >>>>> than

Re: Automatic prefetching in querysets

2017-08-15 Thread Adam Johnson
ing etc. So this is purely about the forward >>>> direction of foreign keys. >>>> >>>> I hope that clarifies my thinking some. >>>> >>>> Regards >>>> G >>>> >>>> On Tue, Aug 15, 2017 at 7:02 PM, Marc T

Re: Automatic prefetching in querysets

2017-08-15 Thread Sean Brant
taining appropriate prefetches and selects is a necessary part of > working with an ORM. > > Do you know of any other ORMs which attempt similar magical optimisations? > How do they go about identifying the cases where it is necessary? > > On 15 August 2017 at 10:44, Gordon

Re: Automatic prefetching in querysets

2017-08-15 Thread Luke Plant
they go about identifying the cases where it is necessary? On 15 August 2017 at 10:44, Gordon Wrigley <gordon.wrig...@gmail.com <mailto:gordon.wrig...@gmail.com>> wrote: I'd like to discuss automatic prefetching in querysets. Specifically automatically doing prefetch_related

Re: Automatic prefetching in querysets

2017-08-15 Thread Josh Smeaton
gt;>> >>>> I'm not a fan of adding a layer that tries to be this clever. How would >>>> possible prefetches be identified? What happens when an initial loop in a >>>> view requires one prefetch, but a subsequent loop in a template requires >>>> som

Re: Automatic prefetching in querysets

2017-08-15 Thread Adam Johnson
gt; possible prefetches be identified? What happens when an initial loop in a >>> view requires one prefetch, but a subsequent loop in a template requires >>> some other prefetch? What about nested loops resulting in nested >>> prefetches? Code like this is almo

Re: Automatic prefetching in querysets

2017-08-15 Thread Gordon Wrigley
ut nested loops resulting in nested >> prefetches? Code like this is almost guaranteed to break unexpectedly in >> multiple ways. Personally, I would argue that correctly setting up and >> maintaining appropriate prefetches and selects is a necessary part of >> working with an

Re: Automatic prefetching in querysets

2017-08-15 Thread Gordon Wrigley
tly setting up and >> maintaining appropriate prefetches and selects is a necessary part of >> working with an ORM. >> >> Do you know of any other ORMs which attempt similar magical >> optimisations? How do they go about identifying the cases where it is >&g

Re: Automatic prefetching in querysets

2017-08-15 Thread Tom Forbes
bout identifying the cases where it is necessary? > > On 15 August 2017 at 10:44, Gordon Wrigley <gordon.wrig...@gmail.com> > wrote: > >> I'd like to discuss automatic prefetching in querysets. Specifically >> automatically doing prefetch_related where needed without th

Re: Automatic prefetching in querysets

2017-08-15 Thread Gordon Wrigley
ays. Personally, I would argue that correctly setting up and >>> maintaining appropriate prefetches and selects is a necessary part of >>> working with an ORM. >>> >>> Do you know of any other ORMs which attempt similar magical >>> optimisat

Re: Automatic prefetching in querysets

2017-08-15 Thread Collin Anderson
rectly setting up and >> maintaining appropriate prefetches and selects is a necessary part of >> working with an ORM. >> >> Do you know of any other ORMs which attempt similar magical >> optimisations? How do they go about identifying the cases where it is >&g

Re: Automatic prefetching in querysets

2017-08-15 Thread Gordon Wrigley
any other ORMs which attempt similar magical optimisations? > How do they go about identifying the cases where it is necessary? > > On 15 August 2017 at 10:44, Gordon Wrigley <gordon.wrig...@gmail.com> > wrote: > >> I'd like to discuss automatic prefetching in querysets. Spec

Re: Automatic prefetching in querysets

2017-08-15 Thread Marc Tamlyn
attempt similar magical optimisations? How do they go about identifying the cases where it is necessary? On 15 August 2017 at 10:44, Gordon Wrigley <gordon.wrig...@gmail.com> wrote: > I'd like to discuss automatic prefetching in querysets. Specifically > automatically doing prefetch_r

Automatic prefetching in querysets

2017-08-15 Thread Gordon Wrigley
I'd like to discuss automatic prefetching in querysets. Specifically automatically doing prefetch_related where needed without the user having to request it. For context consider these three snippets using the Question & Choice models from the tutorial <https://docs.djangoproject.com/