Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  Adam
 |  Johnson
 Type:  New feature  |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by şuayip üzülmez):

 * cc: şuayip üzülmez (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c81781f7e-7dba12e1-0bf0-449f-b32a-0123e299b281-00%40eu-central-1.amazonses.com.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  Adam
 |  Johnson
 Type:  New feature  |   Status:  assigned
Component:  Database layer   |  Version:  dev
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Adam Johnson):

 * owner:  nobody => Adam Johnson
 * needs_docs:  1 => 0
 * has_patch:  1 => 0
 * status:  new => assigned


Comment:

 I’m working on a PR for Django core now, based on
 [https://github.com/django/django/pull/16090/files Andreas Pelme’s
 recently-closed PR] and discussions with Andreas and Simon Charette.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018c17008b60-db02e0c9-01a2-4e0c-94b6-402bf80816f4-00%40eu-central-1.amazonses.com.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  1
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Gordon Wrigley):

 My existing code for this is now available as a pypi package
 https://github.com/tolomea/django-auto-prefetch

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.603174cbc5939cc00f21c11d67bc3a13%40djangoproject.com.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  1
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Tim Graham):

 * needs_docs:  0 => 1
 * has_patch:  0 => 1
 * stage:  Unreviewed => Accepted


Comment:

 [https://github.com/django/django/pull/9064 PR]

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.5d2c36ebaadc2770b4bebc457e73bcad%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Jonas Haag):

 * cc: Jonas Haag (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.a09edf2ba2f0cc9f6d6f5ed443c5ed9e%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Ed Morley):

 * cc: Ed Morley (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.5bf0d0234a0538b8aa7a2e13d67a4730%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Description changed by Gordon Wrigley:

Old description:

> When accessing a 2one field (foreign key in the forward direction and
> one2one in either direction) on a model instance, if the fields value has
> not yet been loaded then Django should prefetch the field for all model
> instances loaded by the same queryset as the current model instance.
>
> There has been some discussion of this on the mailing list
> https://groups.google.com/forum/#!topic/django-developers/EplZGj-ejvg
>
> Currently when accessing an uncached 2one field Django will automatically
> fetch the missing value from the Database. When this occurs in a loop it
> creates 1+N query problems. Consider the following snippet:
>
> {{{#!python
> for choice in Choice.objects.all():
> print(choice.question.question_text, ':', choice.choice_text)
> }}}
>
> This will do one query for the choices and then one query per choice to
> get that choice's question.
> This behavior can be avoided with correct application of prefetch_related
> like this:
>
> {{{#!python
> for choice in Choice.objects.prefetch_related('question'):
> print(choice.question.question_text, ':', choice.choice_text)
> }}}
>
> This has several usability issues, notably:
> * Less experienced users are generally not aware that it's necessary.
> * Cosmetic seeming changes to things like templates can change the fields
> that should be prefetched.
> * Related to that the code that requires the prefetch_related (template
> for example) may be quite removed from where the prefetch_related needs
> to be applied (view for example).
> * Subsequently finding where prefetch_related calls are missing is non
> trivial and needs to be done on an ongoing basis.
> * Excess fields in prefetch_related calls are even harder to find and
> result in unnecessary database queries.
> * It is very difficult for libraries like the admin and Django Rest
> Framework to automatically generate correct prefetch_related clauses.
>
> The proposal is on the first iteration of the loop in the example above,
> when we first access a choice's question field instead of fetching the
> question for just that choice, speculatively fetch the questions for all
> the choices returned by the queryset.
> This change results in the first snippet having the same database
> behavior as the second while reducing or eliminating all of the noted
> usability issues.
>
> Some important points:
> * 2many fields are not changed at all by this proposal as I can't think
> of a reasonable way of deciding which of the many to fetch
> * Because these are 2one fields the generated queries can't have more
> result rows than the original query and may have less.
> * This feature will never result in more database queries.
> * It will not change the DB behavior of code which is full covered by
> prefetch_related (and select_related) calls at all.
> * This will inherently chain across relations like
> choice.question.author, the conditions above still hold under such
> chaining.
> * It may result in larger data transfer between the database and Django
> in some situations.
>
> On that last point an example would be this:
> {{{#!python
> qs = Choice.objects.all()
> list(qs)[0].question
> }}}
> Such examples generally seem to be rarer and more likely to be visible
> during code inspection (vs {{choice.question}} in a template). And larger
> queries are usually a better failure mode than producing hundreds of
> queries.
> For this to actually produce inferior behavior in practice you need to:
> a. fetch a large number of choices
> b. filter out basically all of them
> c. in a way that prevents garbage collection of the unfiltered ones
> If any of those aren't true then automatic prefetching will still produce
> equivalent or better database behavior than without.
>
> Several optin/optout options were discussed in the mailing list, I will
> attempt to summarize these below. Most of them are compatible with each
> other, however in the interests of having a clean interface we probably
> want to limit how many we implement.
> 1. A global option in settings. So as to not accidentally fix existing
> 

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Ryan Hiebert):

 * cc: Ryan Hiebert (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.fe27cb995ce6d5d51d6890768cfef8a6%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Gordon Wrigley):

 For curiosity sake I tried running the test suite with
 auto_prefetch_related enabled by default. There were 3 test failures, two
 were looking for queries that are removed by auto_prefetch_related.
 The third attempts to fetch more rows than the sqlite backend can handle
 in a single 'in' clause, which I'd think is an issue with the 'in'
 implementation.
 Looking at the test it is currently unintentionally doing some four and a
 quarter thousand DB queries. And attempting to fix it with an explicit
 prefetch fails in the exact same manner as the automatic prefetch.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.731d8d0558c3960c85d425590688a488%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Gordon Wrigley):

 Since there was some discussion over optin / optout strategies I have for
 the moment gone with one that seems safe and easy to implement. So
 currently the feature is off by default and enabled by calling
 auto_prefetch_related() on a queryset.

 Related to that I have not addressed documentation at all.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.dcd96e98a4b8cead50baab00c884fc1c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Adam (Chainz) Johnson):

 * cc: Adam (Chainz) Johnson (added)


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.c0ebd5e4fec4a48d791ec31d6229603b%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Django] #28586: Automatically prefetch related for "to one" fields as needed.

[Django]

#28586: Automatically prefetch related for "to one" fields as needed.
-+-
 Reporter:  Gordon Wrigley   |Owner:  nobody
 Type:  New feature  |   Status:  new
Component:  Database layer   |  Version:  master
  (models, ORM)  |
 Severity:  Normal   |   Resolution:
 Keywords:  prefetch_related | Triage Stage:
 |  Unreviewed
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Gordon Wrigley):

 I hope to have a first version of a pull for this up tomorrow

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To post to this group, send email to django-updates@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/065.bd204b8890fcf87d82ff2add020c5629%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.