Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by bcail):

 @HeejunShin, you can try running the example from the ticket Description -
 see if that works with the main branch. I don't think this issue has been
 fixed, though.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018e2e21b07f-2fb78502-7142-4ad3-a911-7bc3993033f6-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by HeejunShin):

 Thank you for answer.
 I'll consider it further.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ddce16f88-13bb2a3e-c67c-4fbb-9e53-6f17427ad336-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by David Sanders):

 You're welcome to try with the disclaimer that you understand what Simon
 has said above about it being quite hard and it will take an in-depth
 understanding of how expressions are resolved & compiled :)
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018dd56bb3cf-2c556262-4fe8-47ee-926b-f550a1056608-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by HeejunShin):

 could i try fix this ticket?
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018dd5301137-d2b3baaf-45a8-484f-9621-18b799ba72fe-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Shafiya Adzhani):

 * owner:  Shafiya Adzhani => (none)
 * status:  assigned => new

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d443f00f0-b0188a2e-bb6a-42e6-8396-f83d9c2bafb9-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Shafiya
 |  Adzhani
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by Shafiya Adzhani):

 Replying to [comment:10 Simon Charette]:
 > The hard part is here is likely due to two factor
 >
 > 1. `OuterRef` resolving is quite finicky and depends on a specific chain
 of resolving during query composition (subquery is resolved and then the
 outer query is)
 > 2. `order_by` members are unfortunately not resolved at query
 composition time (when `QuerySet.order_by` is called) but
 
[https://github.com/django/django/blob/12ffcfc350a19bbfbc203126a9b6c84b5e0d0ba2/django/db/models/sql/compiler.py#L466
 at query compilation time].
 >
 > The reason for 2. is that `QuerySet.order_by` calls are not additive but
 destructive and that the query composition logic doesn't support a generic
 way of eliding joins that we previously created for some operations.
 >
 > For example, say that you have a query that does `qs =
 Book.objects.order_by("author__name")`. If the order by clause is
 immediately resolved then a `JOIN` to `author` must be created. If the
 same query has then its ordering changed to `qs.order_by("title")` then
 we'd want to clear the join to `author` as it's no longer necessary.
 >
 > In order to avoid implementing the logic to ''unreference'' relations
 the `order_by` method deferred the resolving to the very end of query
 compilation which is a different path than the ''normal'' resolving taking
 place in additive methods such as `annotate` and `filter`.
 >
 > I suspect this ticket will be hard to solve without tackling the large
 problem of compile time resolving of `order_by`.

 Thank you for the pointer! Since this ticket is complicated, I'll leave it
 to someone who is interested in solving this problem.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d443ecffa-c80bf94c-2a7c-4a24-ae70-c01f97da41c0-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Shafiya
 |  Adzhani
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Simon Charette):

 The hard part is here is likely due to two factor

 1. `OuterRef` resolving is quite finicky and depends on a specific chain
 of resolving during query composition (subquery is resolved and then the
 outer query is)
 2. `order_by` members are unfortunately not resolved at query composition
 time (when `QuerySet.order_by` is called) but
 
[https://github.com/django/django/blob/12ffcfc350a19bbfbc203126a9b6c84b5e0d0ba2/django/db/models/sql/compiler.py#L466
 at query compilation time].

 The reason for 2. is that `QuerySet.order_by` calls are not additive but
 destructive and that the query composition logic doesn't support a generic
 way of eliding joins that we previously created for some operations.

 For example, say that you have a query that does `qs =
 Book.objects.order_by("author__name")`. If the order by clause is
 immediately resolved then a `JOIN` to `author` must be created. If the
 same query has then its ordering changed to `qs.order_by("title")` then
 we'd want to clear the join to `author` as it's no longer necessary.

 In order to avoid implementing the logic to ''unreference'' relations the
 `order_by` method deferred the resolving to the very end of query
 compilation which is a different path than the ''normal'' resolving taking
 place in additive methods such as `annotate` and `filter`.

 I suspect this ticket will be hard to solve without tackling the large
 problem of compile time resolving of `order_by`.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d2dd5c06e-5282f246-ac90-4776-a8f5-85ce79cbab34-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Shafiya
 |  Adzhani
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Shafiya Adzhani):

 * owner:  REGNIER Guillaume => Shafiya Adzhani


Comment:

 I will try to improve what can be done from previous PR.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d2c0c8215-dd425e40-7769-4a3c-b942-ab9fd375e88c-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  REGNIER
 |  Guillaume
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * owner:  Umang Patel => REGNIER Guillaume
 * needs_better_patch:  0 => 1


Comment:

 [https://github.com/django/django/pull/16994 PR]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ae74d1b13-52bae6de-1c8a-4e5a-9efd-17e206567824-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Umang
 |  Patel
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Umang Patel):

 * has_patch:  0 => 1


Comment:

 https://github.com/django/django/pull/17326

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ae3ea2d6b-d697cf6d-d89a-4999-9346-095efd4c349c-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Umang
 |  Patel
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Umang Patel):

 * owner:  REGNIER Guillaume => Umang Patel


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ae3aea09d-1285cfe1-e91d-4022-a2e8-291565b2c83b-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  REGNIER
 |  Guillaume
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Jordan Bae):

 * owner:  (none) => REGNIER Guillaume
 * status:  new => assigned


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701885b1ae1e1-8e785dae-b12a-47bc-a724-9ac1ea6d62fb-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  (none)
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Jordan Bae):

 * owner:  Jordan Bae => (none)
 * status:  assigned => new


Comment:

 no worry! i think it's better tyou bring this ticket. i deassign this
 ticket.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701885b1a3cbf-2f10a3b8-66c7-46c7-9913-d98cf87449ec-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Jordan
 |  Bae
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by REGNIER Guillaume):

 I've been working in and out on this for the last three weeks. I'm sorry I
 didn't claim the ticket...

 Here is a branch where the issue is solved:
 https://github.com/Alombaros/django/tree/ticket_34533_3_2

 I'm unsure on how to proceed.
 Previously, in the following query :
 
`MyModel.objects.annotate(foo=Subquery(MyOtherModel.objects.order_by(F("pk")).values("pk")[:1]))`
 The `F("pk")` was resolved as the `pk` of the outer query (`MyModel` in
 this case) which does not seems right to me.
 This was due to the order by expressions not being resolved until the last
 minute.

 I force the resolution when annotating a SubQuery so that the OuterRef can
 do its job but this made so `F` objects are resolved as field of the inner
 query.
 To me, this seems to be closer to the intended behavior because its what
 append when calling `.filter` or `.annotate` but if anyone used the prior
 behavior, thing will break

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701883e41576b-a89f4083-9d21-44ca-bb0c-8ab3f9eff6f3-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  Jordan
 |  Bae
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Jordan Bae):

 * owner:  nobody => Jordan Bae
 * status:  new => assigned


Comment:

 I will try to fix this!

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/010701883d4de0c5-6992e889-6577-4a81-833a-157298555dfb-00%40eu-central-1.amazonses.com.

Re: [Django] #34533: OuterRef not resolved as part oh ORDER BY clause

[Django]

#34533: OuterRef not resolved as part oh ORDER BY clause
-+-
 Reporter:  REGNIER Guillaume|Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  3.2
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  OuterRef, OrderBy| Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Sanders):

 * stage:  Unreviewed => Accepted


Comment:

 Thanks 

 Just FYI this will work (note you must wrap 'pk' with an F):

 {{{
 MyModel.objects.annotate(foo=Subquery(MyModel.objects.annotate(order=F("pk")
 - OuterRef("pk")).order_by("order").values("pk")[:1]))
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070187e171b082-6f0ef0a1-f32b-449b-880b-e36f2997cb93-00%40eu-central-1.amazonses.com.