Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  ontowhee
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * resolution:   => fixed
 * status:  assigned => closed

Comment:

 In [changeset:"66e47ac69a7e71cf32eee312d05668d8f1ba24bb" 66e47ac6]:
 {{{#!CommitTicketReference repository=""
 revision="66e47ac69a7e71cf32eee312d05668d8f1ba24bb"
 Fixed #29725 -- Removed unnecessary join in QuerySet.count() and exists()
 on a many to many relation.

 Co-Authored-By: Shiwei Chen 
 }}}
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018db117e676-0bc6ac95-ade7-42b2-b5b6-374ffca6905b-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  ontowhee
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  1 => 0
 * needs_tests:  1 => 0
 * stage:  Accepted => Ready for checkin

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018db0ecdc78-8ef5003a-c820-4060-842e-83771552c4bb-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  ontowhee
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by ontowhee):

 I have removed the changes involving chaining all().
 [https://github.com/django/django/pull/17811 PR]
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d7ecc4255-69d96393-537e-4735-bf87-5b9911ce227a-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  ontowhee
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Comment (by ontowhee):

 I'm not sure if this is a viable solution. It passing the
 `ManyRelatedManager` to the queryset as an instance variable
 `_many_related_queryset`.

 [https://github.com/django/django/pull/17811 Draft PR]

 > No need to as at that point it will be a QuerySet

 I briefly looked into `QuerySet._query` to see if a new optimized sql
 query could be generated when the `aliasMap` lists certain table types. I
 ended up passing the `ManyRelatedManager` to get something working. I'm
 open to exploring a different approach.
-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d6de84e6f-405949fe-0855-413c-ac7d-7bd7650ae12f-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  ontowhee
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by ontowhee):

 * owner:  Shiwei Chen => ontowhee


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018d36e1e7bf-39ebcb64-629e-45b3-9c1c-f286df0a48a8-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  Shiwei
 Type:   |  Chen
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * needs_better_patch:  0 => 1
 * needs_tests:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070188e135b16b-a9bb9866-7783-4b99-92c2-8bb831cc08ed-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  Shiwei
 Type:   |  Chen
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Shiwei Chen):

 * needs_better_patch:  1 => 0
 * needs_tests:  1 => 0


Comment:

 Updated the patch to properly deal with cases involving custom managers
 and prefetch queries, and implemented test cases for them.

 https://github.com/django/django/pull/16863

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070188a15e1a02-5c357803-0842-4948-b515-35f35007a4c7-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  Shiwei
 Type:   |  Chen
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * needs_tests:  0 => 1


Comment:

 Patch needs a few tweaks to properly disable the custom manager (most
 notably tests). It also lacks coverage for the prefetched case as reported
 by comment:18.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018828048a51-faed048a-bf05-4f9f-928d-b8c0352adca4-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  Shiwei
 Type:   |  Chen
  Cleanup/optimization   |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Shiwei Chen):

 * owner:  oliver => Shiwei Chen
 * status:  new => assigned


Comment:

 Thanks everyone for your insights. Im going to try to implement this
 optimization, accounting for all the issues listed here so far.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/01070188262d1f72-f080cdd1-ac94-4550-bc4c-7ca464767efb-00%40eu-central-1.amazonses.com.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mike Hansen):

 Additionally, the optimization causes a query to be done if the related
 objects have already been prefetched.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.6cc0dadbd18f456156894a6a4fabf8d3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-

Comment (by felixxm):

 Ticket #30325 revealed one more issue, i.e. optimization doesn't work when
 chaining `all()` after reverse M2M relations.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.3ada2747501fe1d3baf02a9821e0d147%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * needs_better_patch:  0 => 1
 * has_patch:  0 => 1


Comment:

 Switched back to ''patch needs improvement'' instead as it can probably be
 adapted to skip the optimization when a custom manager is defined.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.9d874d0cc421245794a7e3485f832e6a%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by felixxm):

 * has_patch:  1 => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.fddc2b20a6c9f64053bdce4325f0e06c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  new
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by felixxm):

 * status:  closed => new
 * resolution:  fixed =>


Comment:

 Solution has been reverted because it caused the inconsistent behavior of
 `count()` and `exists()` on a reverse many-to-many relationship with a
 custom manager (see new tests in
 9ac8520fcde29840a1345be19d80dbda53aa6d03).

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.456b2506e451d16c3d81ac1205a59098%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak ):

 In [changeset:"e8de1cc94c9a251ba7c569468f1e94faf9a05671" e8de1cc9]:
 {{{
 #!CommitTicketReference repository=""
 revision="e8de1cc94c9a251ba7c569468f1e94faf9a05671"
 [2.2.x] Fixed #30325 -- Reverted "Fixed #29725 -- Removed unnecessary join
 in QuerySet.count() and exists() on a many-to-many relation."

 This reverts commit 1299421cadc4fcf63585f2f88337078e43e660e0 due to
 a regression with custom managers.

 Backport of 5f7991c42cff73b6278106d499d719b726f85ead from master
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.e4ab82c81cecbd2de75d86608f731d57%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak ):

 In [changeset:"5f7991c42cff73b6278106d499d719b726f85ead" 5f7991c4]:
 {{{
 #!CommitTicketReference repository=""
 revision="5f7991c42cff73b6278106d499d719b726f85ead"
 Fixed #30325 -- Reverted "Fixed #29725 -- Removed unnecessary join in
 QuerySet.count() and exists() on a many-to-many relation."

 This reverts commit 1299421cadc4fcf63585f2f88337078e43e660e0 due to
 a regression with custom managers.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.7aece01a7bd353a7a5093326ce2bb815%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  closed
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:  fixed
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Tim Graham ):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [changeset:"1299421cadc4fcf63585f2f88337078e43e660e0" 1299421]:
 {{{
 #!CommitTicketReference repository=""
 revision="1299421cadc4fcf63585f2f88337078e43e660e0"
 Fixed #29725 -- Removed unnecessary join in QuerySet.count() and exists()
 on a many-to-many relation.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.99235df6eeae2f0c9f81bd8f9a0619b3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by oliver):

 * needs_better_patch:  1 => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.a2ce28b7747b9ea3a0068d13303e3150%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  1
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * needs_better_patch:  0 => 1
 * needs_tests:  1 => 0


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.28bf67fa4e5652869ceaee5dcc7dd7c1%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Simon Charette):

 Doing it for `exists()` as well makes sense.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.b63a973e5d740e7cad411bde0659bae8%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Tom Forbes):

 Have we considered making this change for `exists()` as well? I'm sure
 this will generate the join in the same way that `count()` does.

 Also somewhat related: https://code.djangoproject.com/ticket/28477

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.8e5c4a5d77158979ddff2d9b18e15bb3%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  1|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by oliver):

 * has_patch:  0 => 1
 * needs_tests:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.0e41d2d0ecac939abc319368f15b586f%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by oliver):

 https://github.com/django/django/pull/10366

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.f5678af330f825b6c531daddab8d458c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by oliver):

 Thanks for your advice.
 I will try to correct it according to your advice soon

 Replying to [comment:2 Simon Charette]:
 > Hello Olivier,
 >
 > I think you should be able to achieve this by defining a `count()`
 method on the dynamic class created by
 `create_forward_many_to_many_manager` by filtering
 `self.through._default_manager` based on `self.instance` and return its
 `count()`.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.00f30d21b3cf03622c56c9c3e15ba217%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * cc: Simon Charette (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.c5ea317d56acae8fe581be14397fdf05%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:   |   Status:  assigned
  Cleanup/optimization   |
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Simon Charette):

 * version:  2.1 => master
 * type:  Uncategorized => Cleanup/optimization
 * stage:  Unreviewed => Accepted


Comment:

 Hello Olivier,

 I think you should be able to achieve this by defining a `count()` method
 on the dynamic class created by `create_forward_many_to_many_manager` by
 filtering `self.through._default_manager` based on `self.instance` and
 return its `count()`.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.b37690fcc0a21f056ddfdd4bb8b19931%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
 Reporter:  Gavin Wahl   |Owner:  oliver
 Type:  Uncategorized|   Status:  assigned
Component:  Database layer   |  Version:  2.1
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:   | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by oliver):

 * status:  new => assigned
 * owner:  nobody => oliver


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/067.b0fcc37548c3277a9b2fa3b4039696c4%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

[Django] #29725: Inefficient SQL generated when counting a ManyToMany

[Django]

#29725: Inefficient SQL generated when counting a ManyToMany
-+-
   Reporter:  Gavin  |  Owner:  nobody
  Wahl   |
   Type: | Status:  new
  Uncategorized  |
  Component:  Database   |Version:  2.1
  layer (models, ORM)|
   Severity:  Normal |   Keywords:
   Triage Stage: |  Has patch:  0
  Unreviewed |
Needs documentation:  0  |Needs tests:  0
Patch needs improvement:  0  |  Easy pickings:  0
  UI/UX:  0  |
-+-
 When calling count() on an unfiltered many to many relation, a useless
 join is included in the SQL that makes it much slower than it should be.
 On my dataset, the difference is 1000ms to 100ms, because an index-only
 scan can be used.

 This is the SQL that is currently generated:

 {{{#!sql
 SELECT COUNT(*) AS "__count"
 FROM "app_foo"
 INNER JOIN "app_foo_bar" ON ("app_foo"."id" = "app_foo_bar"."foo_id")
 WHERE "app_foo_bar"."foo_id" = ?;
 }}}

 This is the SQL that should be generated:

 {{{#!sql
 SELECT COUNT(*) AS "__count"
 FROM "app_foo_bar"
 WHERE "app_foo_bar"."foo_id" = ?;
 }}}

 This optimization can only be applied when there are no filters applied,
 because then the join is used to satisfy the filters. In the no-filters
 case, only the through table needs to be consulted.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/052.570bebabd152aef9a862d8e48cb7fecc%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.