Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2013-07-02 Thread Rimvydas Naktinis
Yup, qs.prefetch_related('best_pizza__toppings__topping_type', 
use_select_related=True) looks good enough for many cases.

We have a lot of cases in the code where this would save many queries.

On Tuesday, November 27, 2012 2:57:16 PM UTC, Anssi Kääriäinen wrote:
>
> On 26 marras, 21:36, tolomea  wrote: 
> > The focus then is on how to let the developer specify what they want in 
> the 
> > same query and what they want in a different query. 
> > Select_related currently adds same query stuff and prefetch_related 
> > adds separate query stuff. So what if we added a way for a 
> select_related 
> > to be bolted onto a prefetch_related. A "base" argument to 
> select_related 
> > seems like a reasonable way of doing this. The original example was: 
> > 
> > > 
> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type') 
> > 
> > We could rewrite it as: 
> > 
> > 
> > 
> > 
> Restaurant.objects.prefetch_related('best_pizza__toppings").select_related(base='best_pizza__toppings",
>  
>
> > "topping_type') 
> > 
> > The prefetch_related can be inferred from the base, so we could shorten 
> it 
> > to: 
> > 
> > > Restaurant.objects.select_related(base='best_pizza__toppings", 
> > 
> > "topping_type') 
> > 
> > Each unique unique base produce a new db query, multiple select_relateds 
> > with the same base are done as one query. 
> > I think that's actually a complete system in itself, prefetch_related is 
> > just shorthand and can be implemented as: 
> > 
> > def prefetch_related(queryset, *args): 
> > for arg in args: 
> > queryset = queryset.select_related(base=arg) 
> > return queryset 
> > 
> > The only quirk comes from the original example again, what does this do? 
> > 
> > 
> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type') 
> > 
> > Tired, but I think there is something to this line of thinking. 
>
> The above could work. I would maybe do it the other way, something 
> along lines: qs.prefetch_related('best_pizza__toppings__topping_type', 
> use_select_related=True). This would tell the prefetching to use 
> automatically select_related for any lookups encountered assuming it 
> is possible to select_related the data. For those lookups which are 
> multijoins prefetch would work as always. If you want to prefetch the 
> best_pizza, but select_related the topping_type then use 
> qs.prefetch_related('best_pizza__toppings').prefetch_related('best_pizza__toppings__topping_type',
>  
>
> use_select_related=True) 
>
> Custom queryset use would also be useful. Maybe you want to order the 
> toppings some other way than the default ordering. If a nice API 
> surfaces for this this would be a very nice addition. 
>
>  - Anssi 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2013-02-06 Thread Marc Tamlyn
A better way of achieving this functionality would be the API proposed 
in https://code.djangoproject.com/ticket/17001

Marc

On Wednesday, 6 February 2013 00:29:15 UTC, Michal Novotný wrote:
>
> Hey, what about this 
> one: qs.prefetch_related('best_pizza__toppings__topping_type_*')
>
> It is intuitive and simple. But of course, that would mean to add wildcard 
> support everywhere.
>
> On Sunday, February 19, 2012 11:18:20 PM UTC+1, Yo-Yo Ma wrote:
>>
>> Speaking with regards to the ORM method documented at 
>>
>> https://docs.djangoproject.com/en/dev/ref/models/querysets/#prefetch-related 
>>
>> Of course, ``prefetch_related`` uses a Pythonic join to attach reverse- 
>> related objects and avoids the N+1 queries problem, which of course is 
>> great. However, if you use it like this: 
>>
>> >>> 
>> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type') 
>>
>> It doesn't seem to take advantage ``select_related`` (assuming that 
>> ``topping_type`` is a ``ForeignKey`` from ``Topping``. It instead 
>> sends a separate query for ``Topping`` and ``ToppingType``, joining 
>> them in Python. Is it feasible to modify ``prefetch_related`` use 
>> ``select_related`` when it encounters a ``ForeignKey``?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2013-02-05 Thread Michal Novotný
Hey, what about this 
one: qs.prefetch_related('best_pizza__toppings__topping_type_*')

It is intuitive and simple. But of course, that would mean to add wildcard 
support everywhere.

On Sunday, February 19, 2012 11:18:20 PM UTC+1, Yo-Yo Ma wrote:
>
> Speaking with regards to the ORM method documented at 
>
> https://docs.djangoproject.com/en/dev/ref/models/querysets/#prefetch-related 
>
> Of course, ``prefetch_related`` uses a Pythonic join to attach reverse- 
> related objects and avoids the N+1 queries problem, which of course is 
> great. However, if you use it like this: 
>
> >>> 
> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type') 
>
> It doesn't seem to take advantage ``select_related`` (assuming that 
> ``topping_type`` is a ``ForeignKey`` from ``Topping``. It instead 
> sends a separate query for ``Topping`` and ``ToppingType``, joining 
> them in Python. Is it feasible to modify ``prefetch_related`` use 
> ``select_related`` when it encounters a ``ForeignKey``?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-11-27 Thread Anssi Kääriäinen
On 26 marras, 21:36, tolomea  wrote:
> The focus then is on how to let the developer specify what they want in the
> same query and what they want in a different query.
> Select_related currently adds same query stuff and prefetch_related
> adds separate query stuff. So what if we added a way for a select_related
> to be bolted onto a prefetch_related. A "base" argument to select_related
> seems like a reasonable way of doing this. The original example was:
>
> > Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')
>
> We could rewrite it as:
>
>
>
> Restaurant.objects.prefetch_related('best_pizza__toppings").select_related(base='best_pizza__toppings",
> "topping_type')
>
> The prefetch_related can be inferred from the base, so we could shorten it
> to:
>
> > Restaurant.objects.select_related(base='best_pizza__toppings",
>
> "topping_type')
>
> Each unique unique base produce a new db query, multiple select_relateds
> with the same base are done as one query.
> I think that's actually a complete system in itself, prefetch_related is
> just shorthand and can be implemented as:
>
> def prefetch_related(queryset, *args):
>     for arg in args:
>         queryset = queryset.select_related(base=arg)
>     return queryset
>
> The only quirk comes from the original example again, what does this do?
>
> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')
>
> Tired, but I think there is something to this line of thinking.

The above could work. I would maybe do it the other way, something
along lines: qs.prefetch_related('best_pizza__toppings__topping_type',
use_select_related=True). This would tell the prefetching to use
automatically select_related for any lookups encountered assuming it
is possible to select_related the data. For those lookups which are
multijoins prefetch would work as always. If you want to prefetch the
best_pizza, but select_related the topping_type then use
qs.prefetch_related('best_pizza__toppings').prefetch_related('best_pizza__toppings__topping_type',
use_select_related=True)

Custom queryset use would also be useful. Maybe you want to order the
toppings some other way than the default ordering. If a nice API
surfaces for this this would be a very nice addition.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-11-27 Thread tolomea
The focus then is on how to let the developer specify what they want in the
same query and what they want in a different query.
Select_related currently adds same query stuff and prefetch_related
adds separate query stuff. So what if we added a way for a select_related
to be bolted onto a prefetch_related. A "base" argument to select_related
seems like a reasonable way of doing this. The original example was:

> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')

We could rewrite it as:

>
Restaurant.objects.prefetch_related('best_pizza__toppings").select_related(base='best_pizza__toppings",
"topping_type')

The prefetch_related can be inferred from the base, so we could shorten it
to:

> Restaurant.objects.select_related(base='best_pizza__toppings",
"topping_type')

Each unique unique base produce a new db query, multiple select_relateds
with the same base are done as one query.
I think that's actually a complete system in itself, prefetch_related is
just shorthand and can be implemented as:

def prefetch_related(queryset, *args):
for arg in args:
queryset = queryset.select_related(base=arg)
return queryset

The only quirk comes from the original example again, what does this do?

Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')

Tired, but I think there is something to this line of thinking.


On Mon, Nov 26, 2012 at 6:13 PM, Luke Plant-2 [via Python] <
ml-node+s6n4997386...@n6.nabble.com> wrote:

> On 20/11/12 12:08, tolomea wrote:
> >>  it should do exactly what the developer asks.
> >
> > How do I, the developer, request that it use select_related for
> topping_type
> > part of the best_pizza__toppings__topping_type example given above?
>
> Sorry for the slow reply. At the moment, there is no way to specify
> that. (Unless you can use a default manager to do it, but that might be
> constrained by other things).
>
> It would be nice to come up with a better API for this kind of thing.
> SQLAlchemy may have some good ideas in the way it does eager loading - I
> believe it has support for equivalents of both select_related and
> prefetch_related.
>
> What I'm saying is that in the absence of that better API, or some way
> of specifying the query you want, we shouldn't guess and convert one
> optimisation into another.
>
> Luke
>
> --
> OSBORN'S LAW
> Variables won't, constants aren't.
>
> Luke Plant || http://lukeplant.me.uk/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to [hidden 
> email].
>
> To unsubscribe from this group, send email to [hidden 
> email].
>
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>
> --
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://python.6.n6.nabble.com/Could-prefetch-related-be-modified-to-utilize-select-related-for-ForeignKey-relations-tp4486127p4997386.html
>  To unsubscribe from Could prefetch_related be modified to utilize
> select_related for ForeignKey relations?, click 
> here
> .
> NAML
>




--
View this message in context: 
http://python.6.n6.nabble.com/Could-prefetch-related-be-modified-to-utilize-select-related-for-ForeignKey-relations-tp4486127p4997399.html
Sent from the django-developers mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-11-26 Thread Luke Plant
On 20/11/12 12:08, tolomea wrote:
>>  it should do exactly what the developer asks.
> 
> How do I, the developer, request that it use select_related for topping_type
> part of the best_pizza__toppings__topping_type example given above?

Sorry for the slow reply. At the moment, there is no way to specify
that. (Unless you can use a default manager to do it, but that might be
constrained by other things).

It would be nice to come up with a better API for this kind of thing.
SQLAlchemy may have some good ideas in the way it does eager loading - I
believe it has support for equivalents of both select_related and
prefetch_related.

What I'm saying is that in the absence of that better API, or some way
of specifying the query you want, we shouldn't guess and convert one
optimisation into another.

Luke

-- 
OSBORN'S LAW
Variables won't, constants aren't.

Luke Plant || http://lukeplant.me.uk/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-11-20 Thread tolomea
>  it should do exactly what the developer asks.

How do I, the developer, request that it use select_related for topping_type
part of the best_pizza__toppings__topping_type example given above?



--
View this message in context: 
http://python.6.n6.nabble.com/Could-prefetch-related-be-modified-to-utilize-select-related-for-ForeignKey-relations-tp4486127p4996605.html
Sent from the django-developers mailing list archive at Nabble.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-02-20 Thread Anssi Kääriäinen
On Feb 20, 12:18 am, Yo-Yo Ma  wrote:
> Speaking with regards to the ORM method documented 
> athttps://docs.djangoproject.com/en/dev/ref/models/querysets/#prefetch-...
>
> Of course, ``prefetch_related`` uses a Pythonic join to attach reverse-
> related objects and avoids the N+1 queries problem, which of course is
> great. However, if you use it like this:
>
> >>> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')
>
> It doesn't seem to take advantage ``select_related`` (assuming that
> ``topping_type`` is a ``ForeignKey`` from ``Topping``. It instead
> sends a separate query for ``Topping`` and ``ToppingType``, joining
> them in Python. Is it feasible to modify ``prefetch_related`` use
> ``select_related`` when it encounters a ``ForeignKey``?

Basically a good idea. However, the current stage of prefetch_related
development is "wait and see". That is, prefetch_related is new and
there isn't too good knowledge of why and how people will use it. So,
wait a month or two and after 1.4 has been out for a little while
there is much better knowledge of the pain-points.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Could prefetch_related be modified to utilize select_related for ForeignKey relations?

2012-02-19 Thread Yo-Yo Ma
Speaking with regards to the ORM method documented at
https://docs.djangoproject.com/en/dev/ref/models/querysets/#prefetch-related

Of course, ``prefetch_related`` uses a Pythonic join to attach reverse-
related objects and avoids the N+1 queries problem, which of course is
great. However, if you use it like this:

>>> Restaurant.objects.prefetch_related('best_pizza__toppings__topping_type')

It doesn't seem to take advantage ``select_related`` (assuming that
``topping_type`` is a ``ForeignKey`` from ``Topping``. It instead
sends a separate query for ``Topping`` and ``ToppingType``, joining
them in Python. Is it feasible to modify ``prefetch_related`` use
``select_related`` when it encounters a ``ForeignKey``?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.