Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-19 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  closed
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:  prefetch single- | Triage Stage:  Ready for
  valued order_by|  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"f2388a4b73ee74bd077854798a0ac1669d037304" f2388a4]:
 {{{#!CommitTicketReference repository=""
 revision="f2388a4b73ee74bd077854798a0ac1669d037304"
 Fixed #35309 -- Made prefetch clear ordering for single-valued
 relationships.
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e5a49d4d0-e38d3326-2106-4cab-bcb7-84ec8e52678f-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-19 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Ready for
  valued order_by|  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  1 => 0
 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e5a1ebe72-e7c227f5-411d-45b3-a28f-080c2668898a-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-16 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Accepted
  valued order_by|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Laurent Lyaudet:

Old description:

> While the ordering of multi-valued relationships must be preserved when
> prefetching relationships is it unnecessary when using `prefetch_related`
> against single valued relationships.
>
> For example, given the following models
>
> {{{#!python
> class Author(models.Model):
> name = models.CharField(max_length=200)
>
> class Meta:
> ordering = ["name"]
>

> class Book(models.Model):
> title = models.CharField(max_length=200)
> author = models.ForeignKey(Author, related_name="books",
> on_delete=models.CASCADE)
>
> class Meta:
> ordering = ["title"]
> }}}
>
> The ordering of an author's books in
> `Author.objects.prefetch_related("books")` has a significance as multiple
> books might be associated with each authors.
>
> It's not the case for a book's author in
> `Book.objects.prefetch_related("author")` through as the relationship can
> only contain a single author and there is a single way to order the
> members of a singleton.
>
> In other words `sorted([element], key=sort_func)` will result in
> `[element]` for any `sort_func`.
>
> This property holds true for all the single valued relationships that the
> ORM supports (backward and forward 1:1 and forward 1:M) which allows the
> prefetching to elide any predefined ordering safely to avoid an
> unnecessary and possibly expensive ordering defined for the related model
> queryset.
> Currently the prefetch of authors will use the order by and add useless
> charge on the DB server.
> It would be useful to remove this order by.
> #ClimateChangeBrake

New description:

 While the ordering of multi-valued relationships must be preserved when
 prefetching relationships is it unnecessary when using `prefetch_related`
 against single valued relationships.

 For example, given the following models

 {{{#!python
 class Author(models.Model):
 name = models.CharField(max_length=200)

 class Meta:
 ordering = ["name"]


 class Book(models.Model):
 title = models.CharField(max_length=200)
 author = models.ForeignKey(Author, related_name="books",
 on_delete=models.CASCADE)

 class Meta:
 ordering = ["title"]
 }}}

 The ordering of an author's books in
 `Author.objects.prefetch_related("books")` has a significance as multiple
 books might be associated with each authors.

 It's not the case for a book's author in
 `Book.objects.prefetch_related("author")` through as the relationship can
 only contain a single author and there is a single way to order the
 members of a singleton.

 In other words `sorted([element], key=sort_func)` will result in
 `[element]` for any `sort_func`.

 This property holds true for all the single valued relationships that the
 ORM supports (backward and forward 1:1 and forward 1:M) which allows the
 prefetching to elide any predefined ordering safely to avoid an
 unnecessary and possibly expensive ordering defined for the related model
 queryset.
 Currently the prefetch of authors will use the order by and add useless
 load on the DB server.
 It would be useful to remove this order by.
 #ClimateChangeBrake

--
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e492c814d-d1de3d6f-6425-44b4-a00c-f53ef6fb896d-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-16 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Accepted
  valued order_by|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Laurent Lyaudet:

Old description:

> While the ordering of multi-valued relationships must be preserved when
> prefetching relationships is it unnecessary when using `prefetch_related`
> against single valued relationships.
>
> For example, given the following models
>
> {{{#!python
> class Author(models.Model):
> name = models.CharField(max_length=200)
>
> class Meta:
> ordering = ["name"]
>

> class Book(models.Model):
> title = models.CharField(max_length=200)
> author = models.ForeignKey(Author, related_name="books",
> on_delete=models.CASCADE)
>
> class Meta:
> ordering = ["title"]
> }}}
>
> The ordering of an author's books in
> `Author.objects.prefetch_related("books")` has a significance as multiple
> books might be associated with each authors.
>
> It's not the case for a book's author in
> `Book.objects.prefetch_related("author")` through as the relationship can
> only contain a single author and there is a single way to order the
> members of a singleton.
>
> In other words `sorted([element], key=sort_func)` will result in
> `[element]` for any `sort_func`.
>
> This property holds true for all the single valued relationships that the
> ORM supports (backward and forward 1:1 and forward 1:M) which allows the
> prefetching to elide any predefined ordering safely to avoid an
> unnecessary and possibly expensive ordering defined for the related model
> queryset.

New description:

 While the ordering of multi-valued relationships must be preserved when
 prefetching relationships is it unnecessary when using `prefetch_related`
 against single valued relationships.

 For example, given the following models

 {{{#!python
 class Author(models.Model):
 name = models.CharField(max_length=200)

 class Meta:
 ordering = ["name"]


 class Book(models.Model):
 title = models.CharField(max_length=200)
 author = models.ForeignKey(Author, related_name="books",
 on_delete=models.CASCADE)

 class Meta:
 ordering = ["title"]
 }}}

 The ordering of an author's books in
 `Author.objects.prefetch_related("books")` has a significance as multiple
 books might be associated with each authors.

 It's not the case for a book's author in
 `Book.objects.prefetch_related("author")` through as the relationship can
 only contain a single author and there is a single way to order the
 members of a singleton.

 In other words `sorted([element], key=sort_func)` will result in
 `[element]` for any `sort_func`.

 This property holds true for all the single valued relationships that the
 ORM supports (backward and forward 1:1 and forward 1:M) which allows the
 prefetching to elide any predefined ordering safely to avoid an
 unnecessary and possibly expensive ordering defined for the related model
 queryset.
 Currently the prefetch of authors will use the order by and add useless
 charge on the DB server.
 It would be useful to remove this order by.
 #ClimateChangeBrake

--
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e48c95662-589e23b4-843a-40c6-936c-07501ca83631-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-16 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Accepted
  valued order_by|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Laurent Lyaudet):

 Hello,


 Thanks for the replies and the constructive suggested improvements.


 I removed `_do_not_modify_order_by`.


 I improved to avoid an useless cloning of `QuerySet`.
 For that I created a protected method
 `QuerySet._in_place_empty_order_by()` just below `QuerySet.order_by()`.
 I thought it would be the best solution without modifying the "public API"
 of `QuerySet`.
 Another solution could be to add kwargs to `QuerySet.order_by()` like :
 - `clone: bool = True`, or `no_clone: bool = False`, or `in_place: bool =
 False`,
 - and maybe `clear_default: bool = False`.
 But it would require a larger discussion I think.
 If you think the name of `QuerySet._in_place_empty_order_by()` should be
 changed for something else, tell me.


 Regarding the tests, thanks for the suggestions.
 I applied them in my two tests.
 I think this is better practice to have two separate tests where the
 intent of each test is clear.
 And I haven't added the asserts in other tests.
 However, I understand that you may want to avoid cluttering the tests and
 the models in `tests/prefetch_related/models.py`.
 I have see that I can replace the class `A35309` with the class `Book` and
 the class `B35309` with the class `Author` in my first test.
 Tell me if you think I should do that.
 I have see that I can replace the class `A35309` with the class `Room` and
 the class `C35309` with the class `House` in my second test.
 Tell me if you think I should do that.

 Best regards,
  Laurent Lyaudet
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e48962424-7674d820-9786-4069-98ae-9f5ff2518da9-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships

2024-03-16 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  Laurent
 Type:   |  Lyaudet
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Accepted
  valued order_by|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * owner:  nobody => Laurent Lyaudet
 * status:  new => assigned

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e47ba6f41-cb9b61b8-3e0c-4f30-9eff-2d2f72cd316b-00%40eu-central-1.amazonses.com.


Re: [Django] #35309: Elide ordering of prefetch querysets for single valued relationships (was: Remove Order by on models when prefetching by id)

2024-03-16 Thread Django
#35309: Elide ordering of prefetch querysets for single valued relationships
-+-
 Reporter:  Laurent Lyaudet  |Owner:  nobody
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch single- | Triage Stage:  Accepted
  valued order_by|
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * cc: Simon Charette (added)
 * keywords:  prefetch order_by => prefetch single-valued order_by
 * needs_better_patch:  0 => 1
 * resolution:  wontfix =>
 * stage:  Unreviewed => Accepted
 * status:  closed => new
 * summary:  Remove Order by on models when prefetching by id => Elide
 ordering of prefetch querysets for single valued relationships


Old description:

> Hello,
>
> I don't know if the "bug" is still here with Django 5.
> But on my version of Django, I have the following "bug" :
> Assume you have the following code :
>
> {{{#!python
> class A(models.Model):
> name = models.CharField(max_length=200)
>
> class Meta:
> ordering = ["name"]
>

> class B(models.Model):
> a = models.ForeignKey(A, related_name="bs", on_delete=models.CASCADE)
>

> a1 = A.objects.create(name="a1")
> a2 = A.objects.create(name="a2")
> a3 = A.objects.create(name="a3")
> a4 = A.objects.create(name="a4")
> a5 = A.objects.create(name="a5")
> a6 = A.objects.create(name="a6")
> a7 = A.objects.create(name="a7")
>
> b1 = B.objects.create(a=a1)
> b2 = B.objects.create(a=a2)
> b3 = B.objects.create(a=a3)
> b4 = B.objects.create(a=a4)
> b5 = B.objects.create(a=a5)
> b6 = B.objects.create(a=a6)
> b7 = B.objects.create(a=a7)
>
> bs = B.objects.all().prefetch_related("a")
> }}}
>
> The prefetch of as will use the order by and add useless charge on the DB
> server.
> There may be other cases than ForeignKey where the order by is useless.
> But since OneToOne inherits from ForeignKey, I don't see anything else
> right now.
>
> Hence, I request this enhancement, please :)
> #ClimateChangeBrake
>
> Best regards,
> Laurent Lyaudet

New description:

 While the ordering of multi-valued relationships must be preserved when
 prefetching relationships is it unnecessary when using `prefetch_related`
 against single valued relationships.

 For example, given the following models

 {{{#!python
 class Author(models.Model):
 name = models.CharField(max_length=200)

 class Meta:
 ordering = ["name"]


 class Book(models.Model):
 title = models.CharField(max_length=200)
 author = models.ForeignKey(Author, related_name="books",
 on_delete=models.CASCADE)

 class Meta:
 ordering = ["title"]
 }}}

 The ordering of an author's books in
 `Author.objects.prefetch_related("books")` has a significance as multiple
 books might be associated with each authors.

 It's not the case for a book's author in
 `Book.objects.prefetch_related("author")` through as the relationship can
 only contain a single author and there is a single way to order the
 members of a singleton.

 In other words `sorted([element], key=sort_func)` will result in
 `[element]` for any `sort_func`.

 This property holds true for all the single valued relationships that the
 ORM supports (backward and forward 1:1 and forward 1:M) which allows the
 prefetching to elide any predefined ordering safely to avoid an
 unnecessary and possibly expensive ordering defined for the related model
 queryset.

--
Comment:

 I'm sorry for the awkward back and forth here but reviewing Laurent's PR
 made something clear to me that wasn't from the origin report.

 The requested optimization here is solely for single valued relationships
 (backward and forward 1:1 and forward 1:M). In this scenario, as pointed
 out by Laurent, `ORDER BY` doesn't matter as the related collection is
 either empty or a singleton and thus `order_by()` can always be used in
 their respective `get_prefetch_queryset`.

 In the light of this realization I've adjusted the report and moved back
 this ticket to an accepted optimization.

 Laurent, as for the patch I suggest simply decorating existing tests that
 make use of prefetching for single valued relationship (there are plenty
 in `prefetch_related` tests`) which `assertNumQueries` and use the context
 queries to assert against the lack of `ORDER BY`.

 e.g.

 {{{#!python
 with self.assertNumQueries(2) as ctx:
list(Book.objects.prefetch_related("author"))
 self.assertNotIn("ORDER BY",